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Executive Summary 


1. Purpose of the evaluation 


In February 2019, Promoting Equality in African Schools (PEAS) launched the Inspect and Improve 
Pilot Project (I&l) in partnership with Uganda’s Directorate of Education Standards (DES) under the 
Ministry of Education and Sports (MoES). 


The project aimed to: 


a) Improve the quality of leadership and management in 10 government schools in Eastern 
Uganda, who have been identified as requiring support 


b) Better understand what approaches can improve school quality, how much these cost to 
deliver, and how, if effective, these could be rolled out by MoES at a larger or national scale 


c) Learn about how PEAS can work effectively with non-PEAS schools to promote school- 
and system-level improvements in education quality. 


The pilot ran from Term 1 of the 2019 school year but was interrupted by the global outbreak of 
COVID-19. The project concluded upon the partial re-opening of schools in Uganda towards the 
end of 2020. 


Alongside the implementation of the I&I pilot, an independent evaluation was conducted by 
National Foundation for Educational Research (NFER). The evaluation focused on the second 
objective above, to generate evidence about the impact of I&l and to understand if and how the 
PEAS model of schools inspection and improvement could be successfully implemented in 
government-run schools. This report summarises the findings from the pilot evaluation. 


2. Evaluation design 


The evaluation is based on a mixed-methods comparative case study approach. We used an 
adapted version of the World Management Survey (WMS) to measure project outcomes; the WMS 
has been previously adapted for use in both the education sector and Ugandan context. 


We collected the baseline measure of the WMS in February — March 2019 and endline in February 
— March 2021 with all ten participating intervention schools. We asked respondents to reflect on 
the changes in their practices between the start of pilot and prior to the COVID-19 related school 
closures in March 2020. 


We examined these changes in management practices in greater detail through the use of 
comparative case studies, which explored the perspectives of school leaders, school staff, and 
members of Board of Governors (BoGs) in four of the ten participating schools. We purposively 
selected schools to include those who experienced a greater than average improvement in school 
management practices, and those with less than average improvement as measured by the WMS. 
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Due to the design of our evaluation as well as the adaptations required to respond to changes in 
programming as a result of COVID-19, it should be stressed that the evaluation is unable to directly 
attribute any impacts explored in the findings solely to the I&l pilot. However, where possible, we 
have been systematic in our triangulation of our findings in order to draw conclusions on changes 
to management practices and to elucidate the links between !&! and the observed changes in 
WMS scores. 


3. Findings 


At endline, the strongest management practices demonstrated in pilot schools were related 
to the areas of monitoring and people management 


Our evaluation examined school leadership and management practices prior to the start of the I&l 
pilot and after one year of participation in the I&I pilot project (and prior to COVID-19 related school 
closures), using the WMS. We found that at endline, schools demonstrated strong practices in 
monitoring (including continuous improvement and performance tracking) and people management 
(related to the management of teaching staff). On both, schools scored an average score of 3.9, 
indicating that at endline, school leaders were able to explain or demonstrate formal management 
processes in these areas. 


Compared with WMS baseline measures, pilot schools had the greatest improvements in 
target setting and people management practices and the lowest improvement in leadership 


Our evaluation found all ten participating schools had improved their practices from baseline to 
endline in all five WMS areas (operations, target setting, leadership, as well as monitoring and 
people management). Schools demonstrated particular strengths in the area of monitoring, in 
which school leaders had the highest scores at baseline (2.9) and at endline (3.9) relative to other 
areas. Schools demonstrated the greatest improvements in the areas of targeting setting (average 
score +1.5) and people management (average score +1.9). In both these cases, these 
improvements in WMS scores indicate that schools went from only having some informal 
management practices in these areas to formalised practices. However, leadership practices 
remained the least affected management area, owing to the relative strength of leadership 
practices such as school vision prior to the start of the pilot, or to the relative lack of autonomy to 
change elements such as wider governance structures. 


1&l helped school leaders understand the scope of their role and develop skills to 
successfully monitor teaching and learning 


The evaluation found that with the introduction of I&l, there have been observable changes in 
administrators’ approach and frequency of school and class supervision. School leaders now 
assume an active role in tracking the progress made on the improvement objectives, which implies 
closer supervision of teachers and students and stronger internal monitoring tools and processes. 
Respondents largely agreed that I&I activities helped not only to develop the supervision skills of 
head teachers but also shed light on the role of teachers in the school’s improvement process. The 
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direct impact of the pilot on monitoring processes at two levels (administration and teaching) 
appears to be one of the main catalysts of further changes in case study schools. 


School leaders were able to improve target setting, which resulted in better resource 
prioritisation, accountability and transparency 


Reflected in both the WMS scores and in case studies, improvements to target setting were largely 
driven by improvements in budgeting practices. Case study respondents reported that school 
leaders had improved their financial management skills, which also allowed them to be more 
transparent in, and accountable for, resource allocation decisions. 


1&l helped school leaders enhance people management skills, improving communication 
and working relationships between administrators and staff 


Examining changes to people management, case study respondents reported positive 
improvements in the ways in which school leaders were able to manage school staff by playing a 
more active role in supervising teachers and students and improving internal monitoring tools and 
processes. School leaders played a greater role in driving school improvement processes in terms 
of setting and tracking progress against improvement objectives. School leaders also improved 
communication and working relationships with school staff. This supported their ability to provide 
guidance and direction for improved practices — for example in the use of classroom observations 
as a tool to support improvements in teaching practices. Improvements in communications also 
supported stronger relationships between school leaders and BoGs, as well as with DES 
inspectors. 


The School Improvement Plan (SIP) emerged as the most effective form of support and 
became a management tool for school leaders 


School leaders were able to use SIPs as a management tool to track progress against objectives 
set as a result of initial school inspections and were able to effectively operationalise suggested 
strategies. SIPs were regarded as dynamic rather than stationary, which in turn increased school 
leaders’ ownership of supervision and monitoring processes at the school. The SIP also 
contributed to changing the perception that school leaders and teachers had of the inspection 
process. School leaders therefore have come to see the inspection not an end in itself, but as the 
starting point of self-directed monitoring activities in the school. 


School leaders perceived support and monitoring visits as key to school improvement and 
believed that schools can benefit from more frequent visits 


There was also consensus across case study respondents on the importance of regular support 
visits to follow up on SIP progress and to build a comprehensive picture of the school’s 
environment, including character traits, leadership/management styles and ways of 
communication. This took the form of school visits from PEAS inspection team as well as PEAS 
peer head teachers. Although stakeholders viewed this mechanism as particularly resource- 
intensive, it was recognised as an important driver for change. 


In exploring what facilitates changes in school leadership and management practice, we 
identified two key enablers: 
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Strong working relationships and individual motivation underpin the capacity to 
adopt change in leadership and management practice. This included strong 
collaboration between school leaders and teachers as well as with external school 
stakeholders such as governing bodies, parents and the wider communities. This was 
related to leadership styles adopted by school leaders, whereby schools with the greatest 
changes in WMS scores were also those whose school leaders were associated with more 
transparent and open leadership styles. 


Government schools in the pilot trust the image of PEAS and DES as capacity 
builders and school management experts. The positive perceptions towards both DES 
as the duty bearer for school improvement and PEAS as experts in school management 
supported school leaders’ willingness to participate in the pilot activities and in turn drove 
changes in individual practice. 


While a diverse range of challenges were reported by respondents, two key barriers were 
revealed as factors that inhibit improvements in school leadership and management 
practice: 


Regular head teacher transfers and change in staff hinders the continuity in 
leadership and management reform processes. Amongst the pilot schools, there was a 
change in school leader in four of the ten schools. High turnover rates amongst both school 
leaders and staff meant that it was difficult to maintain the continuity of relationships, as 
well as leadership style and attitude. This was not an insurmountable barrier in all pilot 
schools; however, given the importance of school leaders’ interpersonal relationships and 
attitudes to school improvement, this has the potential to act as a significant barrier. 


Lack of autonomy over people management and limited engagement with governing 
bodies emerged as key barriers to improvement. Some factors were beyond the control 
of individual school leaders to affect, specifically in managing the performance of teachers 
on government payroll. In Uganda, a centralised statutory body, the Education Service 
Commission, is responsible for recruitment, termination, promotion, performance 
management of teaching and non-teaching staff in secondary schools. Where staffing gaps 
exist, school leaders collaborate with Parent Teacher Associations (PTAs) to recruit and 
manage additional teachers. Individual school leaders’ autonomy over people management 
is limited to this category of school personnel. 


The emergence of the COVID-19 pandemic during the pilot created a further set of 
challenges to improvements to school leadership and management for pilot schools in two 


ways: 


COVID-19 disruptions presented challenges to keep school leaders engaged with 1&I 
activities. Much of the progress made by schools prior to the outbreak of COVID-19 was 
effectively halted during school closures. Schools deferred the implementation of their SIPs 
and PEAS was unable to continue with support visits. However, 40 percent of school 
leaders reported that despite this, the skills they developed with the support of I&! proved 
helpful in managing schools during the pandemic. This included situations in which school 
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leaders had strong relationships with governing bodies, parents and communities, as well 
as having a clear idea of existing school challenges through the SIP. 


e Keychallenges to teaching and learning changed from baseline to endline, mainly 
due to the effect of COVID-19 induced disruptions. School leaders reported that budget 
and funding concerns were exacerbated by COVID-19 and related school closures. This 
had a further effect on teacher motivation as school leaders reported difficulties in paying 
salaries of teachers not employed by the government. Student well-being was an additional 
increasing concern. School leaders also reported greater concerns over absenteeism, in 
particular of girls, following the re-opening of schools. 


School leaders perceived that improvements in leadership and management -as a result of 
1&l- have positively affected other areas of schools’ operations 


School leaders reported the changes observed in the seven areas of school operations assessed 
in the DES inspection tool. Unsurprisingly, school leaders reported positive changes in all areas; 
however, improvements in teaching quality and teaching attendance emerged as the most 
frequently mentioned observed change. Schools who had a higher average WMS score at baseline 
were the schools who reported greater changes in school operations at endline. Accounting for 
potential response biases, this suggests that schools who started with a higher level of school 
management practices were able to advance other areas of school operations to a greater extent. 


Additionally, we found that school leaders and staff valued different types of operational changes 
depending on how established their existing management practices were. Schools with lower WMS 
scores focused their responses on the drivers of |&l value on back-end aspects such as ways of 
working and lines of accountability in the school, relationships with administrators, decision making 
and problem solving processes. On the other hand, responses from schools with higher WMS were 
more focused around front-end impact, for example teacher motivation, lesson planning, teaching 
practices, non-corporal disciplinary methods, and better communication between teachers and 
students. 


Pilot schools reported improvements in teaching quality, teacher and student attendance, 
and in learner safety and well-being 


Respondents reported noticeable improvements in the quality of pedagogical practice, lesson 
preparation, development of schemes of work and syllabus coverage. Respondents attributed 
these changes to increased supervision support by school leaders. Teachers also reported 
changes in their peers’ attitude towards classroom observation. With the shift to a more 
participatory and practical pedagogical style, respondents from multiple schools also observed a 
subsequent improvement in the relationship between teachers and students. 


Increased teacher attendance and student attendance emerged as the second most observed 
change by respondents. Case study interviewees attributed these changes to increased monitoring 
support processes introduced by school leaders in the I&I pilot and the adoption of ‘learning walks’ 
with practical observations. 


Finally, 1&I pilot schools also reported increased investment and attention to core areas of learner 
safety within their SIPs. Schools prioritised different issues in the spectrum of learner safety 
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according to their context; for example, some schools focused on reducing the use of physical 
punishment, while others focused on the enhancement of school facilities such as construction of 
school dormitories and fences. School leaders were also able to mobilise parents to co-finance 
school meals in order to support student attendance. Both school-based and stakeholder 
respondents attributed increased learner attendance in these schools to successful roll out of the 
school meals initiative. 


School leaders and stakeholders both felt that improvements brought about by /&l justified 
the costs of PEAS-DES joint inspections approach 


In general, there was consensus amongst stakeholders that the pilot demonstrated value for 
money and that the comparative advantage of the PEAS-DES joint inspection approach is the 
focus on school improvement and follow-up. The I&! approach helped to close the existing gap in 
DES inspection cycle that currently ends with sharing inspection recommendations to schools. 


4. Conclusions 


To understand whether the |&I pilot contains merit for further scaling, we first consider whether the 
pilot has generated sufficient evidence of its effectiveness and impact and an understanding of the 
mechanisms (the project activities and means to trigger change) by which the intervention 
produces change. We then consider three further criteria for scale (relevance, implementation 
quality, and implementation feasibility) to generate lessons from the I&I pilot that may be important 
to consider in the scale-up phase of the I&! project. 


Was I&I successful in improving the quality of leadership and management and why? 


The evaluation found that there is convincing evidence that |&l was successful in improving the 
quality of leadership and management in all ten participating schools across all areas of 
management as defined by the WMS. There is also emerging evidence that improvements to 
school management have led to improvements in student and teacher attendance, teaching 
practices, and student safety and well-being. Evaluation respondents identified SIPs and school 
visits as the two most effective mechanisms from |&l to support improved school management. 


What can we learn from the 1&I pilot on the potential for scale? 


e Engaging school leaders from the start -and with the collaboration of other school 
stakeholders- ensures that SIPs and 1&l support are relevant to school needs. The !&! 
approach was also most successful when it involved cooperation amongst school leaders 
and governing bodies and school staff in order to further tailor the improvement process. 


e Culture and mind set are important factors that require shifting before increased 
school management capacities can result in school improvements. Where school 
leaders held amenable mind sets or where I&l was successful in shifting their mind set 
towards one of ‘improvement’, schools were able to enact wider management 
improvements. 
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e 1&l is a timely intervention that falls within articulated government priorities for 
education reform. The PEAS approach to inspections and school improvement is relevant 
to the strategic direction of government of Uganda, as it puts emphasis on school 
improvement and the subsequent monitoring that goes beyond the inspection visit. Multi- 
stakeholder engagement at national level is required to ensure that school management 
continues to be a policy priority to support improved learning outcomes. 


e The partnership between PEAS and DES in co-creating and implementing the 1&l 
model was crucial to the pilot’s successful adaptation and implementation of 
approaches to inspections and improvements in government schools. The |&I pilot 
appears to have successfully merged PEAS’ model of improvement plan with DES’s newly 
developed inspection tool. The partnership nature of the project that highlights the roles of 
PEAS and DES as co-creators and co-implementers is crucial to reinforce the school 
support network and generate ownership and commitment to the process. 


e One year was sufficient to observe changes to school leaders’ management skills 
and for school leaders and staff to perceive wider changes within their schools. The 
results of both the WMS and case studies confirm that within the pilot timeframes, the pilot 
was successfully able to generate observable changes to both attitudes and mind set, as 
well as practice, in intervention schools. This was particularly evident in school monitoring 
practices, such as standardised supervision processes and practices. 


e High turnover of school leadership and low foundational management capacities are 
related; these factors challenge the extent to which I&l can support school 
improvements. Although the lack of experience or lack of confidence did not impede 
school leaders, and thus schools, from benefitting from the I&I pilot, it may have limited the 
extent to which they were able to benefit as compared to their peers who had more 
established roles and experience. 


e 1&lI has the potential to impact a wider number of schools through indirect impact to 
neighbouring schools. Participating school leaders reported interest and a willingness to 
build on their experience of and participation in I1&l to help support other neighbouring 
schools; their participation in the I&I pilot appears to have provided them with the 
confidence to share learnings with schools that face similar challenges. 


e Although the costs of the 1&l inspection and monitoring process are presumed to be 
higher than the costs of a regular inspection visit, the additional cost is justified by 
the improvements they offer at both the school and systems-level. This includes the 
advantages offered by the joint inspection process for cross-learning and in the 
effectiveness offered to the observations and school inspection visits. 


5. Recommendations 


Finally, our evaluation offers the following recommendations. 
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The expectations of school leaders for the types of support offered by l&I 
requires management. In the next phase, the I&l project should ensure that it 
articulates the types of support that school leaders will receive and the importance and 
value-add of professional development. The first-hand accounts and experience of 
school leaders who participated in the pilot may be useful for this, given their 
enthusiasm for further I&l scale out. 


Some school leaders would benefit from additional time and training on 
foundational management skills in order to maximise on their opportunity to 
learn from their participation in !&!. This training or support could be provided by the 
programme or by bringing in an additional project partner to fill this gap. 


1&l can further its impact by building and nurturing communities of practice as 
part of professional development for school leaders. Where possible, this should be 
built upon existing structures and professional development networks to ensure greater 
sustainability of these relationships. 1&l may want to also explore other approaches to 
facilitate the sharing of experiences and knowledge within and amongst participating 
schools, such as arranging school visits or organising seminars for participants to share 
their successes and challenges. 


1&l should continue to supporting school leaders to focus on learner safety and 
well-being. Given the additional recent experience of school closures and its ancillary 
effects to student safety and well-being, |&l should continue to promote positive 
discipline management and other training focused on learner well-being for future 
support training to school leaders and their staff. 


Although not an explicit focus on of the evaluation, respondents mentioned that 
the development of ICT competencies would be a beneficial addition to the 
professional development of school leaders. I&| may choose to develop and nurture 
partnerships to help support the development of ICT competencies as part of school 
leader professional development. This will also support I&I with the introduction of 
digitised school inspections processes. 


While the pilot has demonstrated that the I&l offers a valuable model for school 
improvement and lessons for I&l scale out, there may need to be a further 
translation of lessons to DES’ own inspections process. There may need to be 
further efforts for DES to institutionalise school improvements within the enhanced 
school inspection approach and a fundamental shift is to advance from focusing on 
publication of inspection reports, to the provision of targeted support for defining and 
monitoring individual school’s-level plans for improvement. 


Given the strong partnership and reputation of PEAS and DES, I&l has an 
enormous potential to leverage this partnership and become an advocacy 
platform to advance system-wide support for school improvement. PEAS and 
DES should continue to find ways to increase exposure to the programme, leveraging 
policy impact and impacts to schools outside the I&! project. In the immediate term, the 
findings and impact of the project should be shared more widely to increase the 
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programme’s visibility and to facilitate embedding the programme into the government 
inspection process. 


8. Through the experience of I&l, DES can play a key role as the champion of the 
programme, channelling evidence of impact to the MoES. DES may be able to 
leverage the evidence from the I&I pilot and beyond to advocate for resources for 
enhanced school inspection process and school improvement, as well as amplifying the 
voice of school leadership to the government. DES can also ensure that issues related 
to school improvement maintain visibility throughout MoES’ policy cycle including the 
next national education sector analysis. 
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1 Introduction 


1.1 Purpose of the Evaluation 


In February 2019, Promoting Equality in African Schools (PEAS) launched the Inspect and Improve 
Pilot Project (I&l) in partnership with Uganda’s Directorate of Education Standards (DES) under the 
Ministry of Education and Sports (MoES). 


The project aimed to: 


e Improve the quality of leadership and management in ten government schools in Eastern 
Uganda, who have been identified as requiring support 


e Better understand what approaches can improve school quality, how much these cost to deliver, 
and how, if effective, these could be rolled out by MoES at a larger or national scale 


e Learn about how PEAS can work effectively with non-PEAS schools to promote school- and 
system-level improvements in education quality. 


The project was intended to run from the start of Term 1 of the 2019 school year (February 2019) 
to the close of Term 1 of the 2020 school year (May 2020); however, the global outbreak of the 
COVID-19 pandemic and resulting school closures in March 2020 created a disruption to the 
project. As a result, 1&l was extended until schools partially re-opened in Uganda in October 2020. 


To support this project, PEAS worked closely with the National Foundation for Educational 
Research (NFER), a UK-based educational research charity. NFER was asked to conduct an 
independent evaluation of the I&I pilot to run alongside implementation. Our evaluation focuses on 
the second objective above to generate evidence about the impact of I&l and understanding if and 
how the PEAS model of school inspection and improvement can be successfully implemented in 
government-run schools. This final report summarises the findings from the pilot evaluation. 


1.2 Structure of the Report 
The structure of the report is as below: 


e Section 2 provides the background to the project, including an exploration of the project’s 
context and design. 


e Section 3 summarises the approach to this evaluation, including details on the evaluation 
design and data collected. 


e Section 4 sets out the findings of the evaluation, roughly organised around the evaluation 
questions. 


e Section 5 provides the key conclusion from the evaluation findings. 


e Section 6 sets out our recommendations for project stakeholders. 
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2 Project Background 


In the baseline report of this evaluation, we explored the project context in order to interrogate the 
1&l Theory of Change (ToC) (see Section 2.2 and Appendix A). This section summarises our 
observations on the project context, including an overview of the Ugandan education system, in 
order to better understand the relevance of I&l to the Ugandan policy context and some of the 
current literature that links schools inspections to improvements in school management and 
accountability. Following this, we provide an overview of the I&l project design and a summary of 
the ToC. 


2.1 Context 


The following section presents an overview of the relevant and recent changes in Uganda’s 
education landscape that help contextualise the motivation for cooperation between PEAS and 
DES for the design of the 1&I pilot. 


Recent government efforts to improve the availability of education in Uganda have been 
successful 


In Uganda, the education system is governed by MoES, who set the standards, provide technical 
guidance and coordinate and monitor the sector. Education is provided for children aged 3-5 years 
old through early childhood programming, which is followed by seven years of compulsory primary 
education. Upon completion of primary education, school leavers are required to complete the 
Primary Leaving Examinations (PLE). Students may then proceed to six years of secondary 
education, with four years of lower secondary and two years of upper secondary. Lower secondary 
school leavers are required to take the Uganda Certificate of Education (UCE) exams and upper 
secondary leavers complete the Uganda Advanced Certificate of Education (UACE) in order to 
proceed to post-secondary education. 


Uganda has experienced an increase in access to education over recent decades, following the 
introduction of Universal Primary Education (UPE) in 1997 and Universal Secondary Education 
(USE) in 2007. In order to increase access to education for Uganda’s rapidly growing population, 
education services are provided either by the government or through private providers, which 
includes community-founded schools, faith-based schools, privately owned schools, and schools 
run under public-private partnerships (PPPs). At the primary level, there are more government 
schools (64 percent) than private (36 percent), but the reverse is true at the secondary level with 
38 percent government and 62 percent private (MoES, 2017). A key part of the Universal 
Secondary education policy was the expansion of public-private partnerships that accelerated 
progress on education access, with secondary school enrolment experiencing an increase of 47 
per cent between 2002 and 2014 (UBOS, 2017). In 2018, the government of Uganda phased out 
USE Public-private partnership scheme. 
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While enrolment in Uganda has increased, challenges related to quality of education persist 


In 2019, primary gross enrolment rates stood above 100 percent' and net enrolment rates were 
close to 90 percent. However, despite gains in enrolment levels, the primary school completion 
rates and primary to secondary transition rates were declining: while lower secondary school 
completion rates had been relatively constant from 2013 to 2017 (ranging from 35 to 38 percent), 
in the same period, transition rates to upper secondary declined from 32 percent to 24.8 percent 
(UBOS, 2019). In the Education sector strategic plan 2017/18-2019/20, the First Lady and Minister 
of Education and Sports reaffirmed this writing in the foreword, ‘The one part that we now must 
strive to address is the quality aspect‘. 


Low transition rates are accompanied by low levels of learning throughout the education system, 
which begins at the primary level. According to data from the National Assessment of Progress in 
Education’s testing of secondary students over time (NAPE, 2014), between 2007 and 2014 a drop 
was recorded in the percentage of Senior 2 students rated as proficient in English Language from 
81.9% to 49.3%. Over the same period, proficiency in mathematics fell from 69.4% to 41.5% and 
proficiency in biology fell from 36.7% to 20.5%. Additionally, in both reading and mathematics, 
Uganda has lagged behind other countries in East Africa region. According to UWEZO (2016), four 
out of ten children in Primary 3 cannot read a word in their local language, while almost one out of 
five children in Primary 3 cannot recognize numbers between 10 and 99. SACMEQ results showed 
that almost 20 percent of Ugandan Primary 6 pupils were performing below the established reading 
benchmarks, while 40 percent of Primary 6 pupils were performing below the mathematics 
benchmark (SACMEQ 2007). 


Interventions focused on improving school management processes have been evidenced 
as a means to address stagnant progress on quality of learning 


Several systematic reviews on what works in education in low- and middle-income (LMIC) 
countries confirm that, while interventions that focus on getting children into school can succeed in 
doing so, there is less evidence to demonstrate that such interventions also prompt improvements 
in learning outcomes (Krishnaratne et al., 2013; Snilstveit et al., 2016). On the other hand, there is 
evidence that interventions targeting various forms of school governance can be effective in 
improving learning outcomes (Evans and Popova, 2015; Glewwe and Muralidharan, 2015; GEC 
2018). 


Here, governance is taken to encompass a wide set of characteristics of education systems and 
how they are managed, including decisions around organisation of instruction, planning and 
structures across the school, decisions on resources and resource allocation, and people 
management (GEC, 2018; Glewwe and Muralidharan, 2015). In both the LMIC and the Ugandan 
context, there is also strong evidence of the positive relationship between strengthening school 
governance through management practices and improved school outcomes (Bloom et al. 2014; 
Crawfurd, 2017). 


' The primary gross enrolment rate is the number of students in primary education divided by the official primary school age population, 
hence when there are repeaters or early/late enrolment, the gross rate can be greater than 100 percent 
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School inspections help produce both vertical accountability (compliance with laws and regulations 
and creating an additional means of assessment for system-wide school performance) as well as 
horizontal accountability (internal school accountability through shared expectations amongst head 
teachers, teachers, students, and other stakeholders) (Eddy-Spicer et a/., 2016). School 
inspections can support the process by which school management is able to support 
improvements to teaching and learning by providing crucial data, feedback, and support in 
diagnosing problems and implementing changes to improve learning outcomes. 


In Ugandan education policy, there is growing recognition of the importance of the role of 
school management and school inspections to improve education quality 


In an effort to address some of these persisting challenges, the 2017-2020 Education Sector 
Strategic Plan (ESSP) set out the Ugandan government’s efforts to shift emphasis from increasing 
the provision of education to implementing more focused and strategic interventions to improve the 
quality and relevance of education (MoES, 2017). In aiming to ensure the delivery of relevant and 
quality education, one of the strategic focuses of the ESSP is to strengthen the current inspection 
system and to establish DES as a semi-autonomous body in charge of inspection, to help improve 
the quality of leadership and management. 


DES was established in 2008 to oversee school inspections and document and share best 
practices in the education system. The ESSP identified low levels of enforcement of 
recommendations from inspection reports as a key weakness in the area of quality assurance. 
Efforts to address this within the ESSP include the mandate to increase the percentage of 
secondary schools with two inspection visits per term from 80 percent of all schools to 100 percent 
by 2020 and the roll out of the new inspections tool which took place in 2019. 


School improvement planning is further recognised by MoES as a means to contribute to 
achievement of national and provincial level targets while achieving their individual targets 
(MOEST Guide 2012). School improvement planning involves a joint school decision making 
process for determining the objectives and activities of a school and the costs involved to ensure 
improvements in teaching and learning. Basic and simple forms of self-evaluation at school-level 
have been emphasised as a feasible and practical way to bring about process of self-reflection and 
school improvement (Scheerens et a/., 2000). 


Efforts to support improved school management in Uganda, in particular through school 
inspections, requires additional support 


In sum, this review suggests that interventions that support improved school management and 
inspections have the potential to improve learning outcomes, and are of relevance in the Ugandan 
context. However, there are a number of challenges and barriers faced by inspection systems that 
may curtail impact. 


Scholars have identified the limited capacity of community bodies to ensure and enforce strong 
school management (Najjumba et a/., 2013) and existing weaknesses in school management 
practices around monitoring teachers and providing them with feedback on teaching practices and 
learning (Sabarwal et al., 2018; Najjumba et a/., 2013). A related challenge has been in the nature 
and quality of feedback provided from an inspection due to the lack of resources afforded to 
inspections, which results in low buy-in and trust from teachers (Hossain, 2017). The relevance of 
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feedback and the demeanour of feedback delivery have been identified as barriers to effectiveness 
of inspections in Uganda (Macharia and Kiruma, 2014). 


Furthermore, inspections have been presented more as a ‘one-off process’, rather than a cycle 
with pre- and post- measures to assess improvement over time. This has resulted in a lack of buy- 
in from key school management stakeholders. Finally, the responsibility to act on inspection 
feedback often lies with the school leader, who often lacks the capacity to do so (Macharia and 
Kiruma, 2014). Both of these challenges highlight the central role played by school leaders in 
enabling the effectiveness of school inspections to promote school improvement. 


2.2 The Design of the I&l Pilot 


In this section, we introduce the I&l partnership and pilot, the theory of change (ToC) that 
underpins its design, and provide an overview of the adaptations made in an evolving operational 
context. 


The conception of the l&I pilot is grounded in previous evidence which suggests that the 
PEAS model of school inspections has contributed to higher learning outcomes for PEAS 
schools 


PEAS is a UK-based non-profit that aims to expand equitable and high quality secondary school 
education in Africa. Since 2009, PEAS has built and run a network of 28 schools in Uganda with 
the aim of providing affordable, high quality and sustainable secondary schools. As a leading 
provider of secondary education in Uganda, PEAS also aims to work with other education 
providers, in particular government partners, to share best practices and pioneer new approaches 
to deliver education that unlocks the full potential of all children. 


A study conducted by the Research on Improving Systems of Education (RISE) programme 
examined school management practices and public-private partnerships (PPP) in Uganda. The 
study found that PEAS schools had higher levels of management quality than comparable 
government, private or other PPP schools, as measured using the World Management Survey 
(WMS) (Crawfurd, 2017). Importantly, the same study showed that these higher levels of 
management quality were associated with higher learning outcomes, via student value-added 
scores. 


A previous external evaluation of PEAS schools identified several aspects of school management 
that has set PEAS’ performance apart from comparable government and private schools. The 
evaluation found that a key factor contributing to the high quality of school management is PEAS’ 
approach to school inspections and accompanying package of support and follow-up. These 
include teacher support and training, accountability measures, child protection, support for learning 
and development of strong school leaders (EPRC, 2018). 


The inputs and activities of 1&l are a package of support mechanisms that are focused 
around the inspections process 


The I&l approach builds on current efforts from DES to strengthen its school inspection approach 
and combines this with PEAS’ school improvement model. The model focuses on building the 
capacity of school leaders to identify improvements required at the school level, centred around the 
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inspections process. In addition to the school inspections, the I&I pilot has developed a package of 
support for school leaders, which includes the development of a School Improvement Plan (SIP) 
on the basis of the initial inspection (See Box 1), continuous support and mentorship throughout 
the school year (in the form of visits and phone calls), culminating in a final inspection to assess 
progress against the SIP and to allow school leaders to reflect on their school’s progress. The 
process is summarised in Table 1 below. 


Box 1. The use of the School Improvement Plan (SIP) as part of the PEAS model 


The SIP is a key leadership and management tool used in PEAS school network. This 
document converts inspection recommendations into specific objectives and strategies for 
improvement. For the I&I pilot, PEAS adapted the SIP to the areas covered by the DES 
inspection tool. After each school inspection, PEAS worked closely with the school leaders to 
identify the weaknesses and priorities of the school, and together set specific targets aligned to 
the DES inspection recommendations and outline strategies for improvement. These strategies 
and measures for success were set for implementation over three academic terms, and 
progress was monitored by the inspection team during their support visits. 


Table 1. Phases and activities of the 1&I pilot 


Phase 
Phase 1: School 
inspections 


Phase 2: Development of 
School Improvement Plan 
(SIP) 


Phase 3: School 
improvement monitoring 
visits 


Phase 4: Final inspection 
and project closure visit 


Activities 

One-day inspection visit by PEAS and DES staff using DES 
inspection tool to identify school strengths and areas for 
development. 


Inspection report produced and findings discussed with 
leadership team. 


SIP development workshop for school leaders. 


Development of the SIP, outlining improvement objectives and 
strategies, signed off by PEAS and DES. 


To support schools to implement their SIP and keep joint track of 
progress: 


e One or two visits per term by PEAS inspector 
e Weekly calls with PEAS school leader mentor 
e One or two observation visits to mentor’s schools 


Postponed due to COVID-19 school closures. PEAS did school 
visits in December 2020 and final school inspections have been 
planned for 2021, though this is likely to face further disruption 
following the re-closure of schools announced in June 2021. 
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Schools were selected by DES for participation in the pilot, on the basis that they required 
improvement, primarily on school leadership and administrative skills. This is particularly true for at 
least half of the schools in the pilot, as these schools were formerly privately-run schools which are 
now managed by the government as seed schools. As such, in previous years these schools had 
not been assessed against the same accountability processes such as inspections or government- 
mandated leadership standards. 


According to the ToC of I&l, as a result of the application of the 1&l model in government 
schools, PEAS and DES will generate observable changes to management and leadership 
practice, which will ultimately result in improved learning quality 


The ToC is explained in detail in the NFER baseline report. The ToC can be summarised as: by 
developing an enhanced model for schools inspection and improvement, the I&! pilot seeks to 
generate observable changes in school management and leadership practices. At the outcome 
level, if |&l improves aspects of leadership quality, this will translate into improved areas of school 
performance, as indicated by inspections results and scores (see Figure 1). 


Figure 1. Expected outputs, outcomes and impact of the I&l Pilot 
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The pilot has a second outcome objective, which is to assess and generate evidence on the 
effectiveness of the developed |&I model to support further scale up and out to government 
schools. 


Finally, if successful, the impact of I&I will be an improvement in teaching quality and learning 
outcomes for students at intervention schools, which can be scaled to reach a greater number of 
schools. This long-term improvement in student outcomes will be beyond the project and 
evaluation’s lifetime to observe and measure. 


While efforts to improve school accountability and management were interrupted due to 
COVID-19, 1&l now has the chance to continue to learn and expand 


In March 2020, schools closed in Uganda in response to the global outbreak of COVID-19. The 
MoES responded with a Preparedness and Response Plan (2020), which outlined various 
measures for an extended lockdown scenario, including: 


e guidance and resources for continued learning, for example, self-study materials and delivery of 
lessons via radio and television programming 


e organisation of accessibility of distance learning resources 


e improving cross-sector communication and coordination between key stakeholders, such as UN 
agencies, NGOs and Ministries. 
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Later measures included revised school calendars and the development of Standard Operating 
Procedures (SOPs), which provided guidelines on the reopening of schools. 


As aresult, the implementation of the I&I pilot was disrupted and experienced a seven month 
pause, starting up again with a partial reopening of schools’ in October 2020. During school 
closures, PEAS staff made efforts to stay in touch with I&l school leaders to offer support and 
prepare for reopening, which was limited by connectivity challenges. Final inspections inspections 
have been planned for 2021, though is likely to face further disruption following the re-closure of 
schools announced in June 2021. 


Despite school closures and subsequent disruption of I&I pilot implementation, the project was able 
to garner buy-in from key government stakeholders and secured funding to expand /&l to a further 
40 government schools across all regions in Uganda and to continue to provide support to the ten 
pilot schools. 
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3 Evaluation Design 


In this section, we provide an overview of our evaluation approach and our data collection methods 
and sampling strategy. We also provide information about how we adapted the evaluation due to 
COVID-19 disruptions. 


3.1 Evaluation Approach 


The overall approach of evaluation remains unchanged from baseline (NFER, 2019). We use a 
mixed-methods comparative case study approach. Using the comparative case study approach 
allows us to compare two or more cases to derive insight into the enablers and barriers of an 
intervention’s success (Goodrick, 2014). We use this approach in order to provide conclusions 
about causal attribution, in the absence of an appropriate use of a comparison group. The 
approach has also allowed us to examine an intervention that has adapted, whether by design or in 
response to a global pandemic. 


Using this approach, our evaluation sought to answer the five evaluation questions in Table 2, 
mapped against three of the OECD-DAC criteria. 


Table 2. Evaluation questions 


OECD-DAC Evaluation question 
criteria 
Effectiveness 1. What impact, if any, did the intervention have on the quality of 


school leadership and management in treatment schools? 


2. Why did the intervention or elements of the intervention 
demonstrate the observed impact? 


3. What other areas of the school's operations, such as - for 
example - staff management or planning processes - if any, did 
the PEAS intervention affect? 


Efficiency 4. How much did the interventions cost to deliver and did they 
represent value for money? 


Sustainability 5. What lessons can be learned from the pilot about how the 
intervention can be scaled out? 


Our evaluation questions are elaborated upon in our evaluation framework (Appendix B), which 
further breaks them into sub-questions, and maps them against the sources of data used for each 
question. 


3.2 Outcome Measure 


The primary outcome measure for this evaluation is an adapted WMS score. The WMS was 
developed in 2002 as a systematic way to measure the quality of management practices, including 
for education management, across the world (Bloom et al., 2014). 
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The WMS involves a structured survey in which trained enumerators generate scores for school 
leadership and management by delivering a set of probing questions to head teachers around 
management practices. We have used the WMS tool that was first developed for use in the 
education sector and adapted for use in Uganda as part of the 2015 RISE Management Survey 
(Crawfurd, 2017). We adapted the basic education WMS for the purposes of our evaluation, 
combining insights offered in the Crawfurd (2017) version with practical considerations around 
time, potential respondent fatigue and the nature of the intervention group. More information about 
our adaptation of the WMS can be found in the baseline report (NFER, 2019). 


The WMS provided the basis for both measuring the pre-post changes in school management 
across all intervention schools as well as for the selection of our comparative case studies. On the 
basis of changes in WMS score from baseline and endline, case studies were selected to further 
explore the two schools who underwent above average improvement in school management, and 
the two schools who experienced below average change. 


3.3 Adaptations to the Evaluation Design 


At baseline, we focused on interrogating the I1&l ToC, and examined the context in which the 
project is situated. We collected baseline WMS scores and analysed the pre-intervention statistics 
for the quality of leadership and management in intervention schools. 


At endline, we intended to collect WMS scores upon the final conclusion of the intervention (the 
final school inspection), in order to track the progress of intervention schools, to draw conclusions 
about the effectiveness, impact and value for money of the I&I pilot, and to generate learning to 
support any future scaling of the pilot. However, in light of the outbreak of COVID-19 and related 
school closures in Uganda, we had to consider a number of adaptations to our evaluation design. 
These included: 


e Changes to the I&I project timelines: The implementation of the pilot was disrupted due to 
school closures. As a result, endline data collection for the I&l pilot evaluation delayed. 
Furthermore, the final school inspection would not be implemented within the evaluation 
timelines and therefore the endline of the WMS does not necessarily reflect the conclusion of 
intervention activities, and therefore the intervention’s ‘dosage’. However, we focus instead on 
generating learnings that can be applied to further project scale up. 


e Changes to outcome measurement: The original purpose of the WMS measures was to 
provide a means to understand if the project had impact and to identify comparative case studies 
for further qualitative investigation. The disruption in implementation limits the extent to which we 
can directly attribute changes between pre-post WMS scores to the I&l pilot. Instead, the 
evaluation explores whether I&l contributed to any observed changes in the quality of school 
management and leadership in intervention schools. 


3.4 Data Collection and Sampling 


We used a mixed-methods data collection design, sequencing quantitative data collection with 
qualitative data collection. Our quantitative methods, used at both baseline and endline, are 
outlined in Table 3 while our qualitative methods, used at endline only, are outlined in Table 4. 
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Table 3. Quantitative data collection methods, used at both baseline and endline 


Tool Description Purpose 
Basic school 15 minute telephone survey To understand school context, school 
survey uptake of pilot activities, effectiveness of 
the pilot, and impact of COVID-19 
Adapted World Structured 60 minute To ascertain the quality of school 
Management telephone survey with 12 leadership and changes over time 
Survey (WMS) questions covering five 


management areas 


Table 4. Qualitative data collection methods, used only at endline 


Tool Description Purpose 
Case study 60-90 minute structured To explore the pilot’s implementation, 
interviews interview with open-ended impact and perceived utility, enablers and 
questions barriers to Success 
Key stakeholder 60 minute structured To explore pilot’s implementation and 
interviews interview open-ended impact, enablers and barriers to success, 
questions value for money and scalability 


considerations 


The evaluation’s data collection tools (WMS and basic school survey, case study and stakeholders’ 
interview guides) can be found in Appendix D. Baseline data collection took place in February and 
March 2019. Endline data collection took place in February and March 2021. All ten government 


S 


econdary schools selected by PEAS and DES to participate in the pilot responded to the basic 


school survey and the WMS at baseline and endline. Case studies were undertaken in four 
schools. The criteria for selection of these schools include: 


Change of WMS scores from baseline to endline: two schools whose changes in WMS 
scores were greater than average and two schools whose changes in WMS were less than the 
average. 


Availability of the school: as fieldwork was carried out during Term 3 and the exam period, 
four head teachers opted out of data collection. 


Input from PEAS: when two or more schools met all the same criteria, we consulted with PEAS 
in order to select schools which were believed to garner strong insights and learnings of the 
pilot's implementation in those schools. 


In each case study school, we conducted five interviews. In order to represent a range of 
perspectives on school management, we selected from amongst the following participants for 
interview: 
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e School leader: The Head Teacher or in their absence, the Deputy Head Teacher 


e School Staff: Two teachers, including men and women and different levels of experience. If 
possible teachers with leadership and management responsibilities. 


e School Management Stakeholders: Two representatives of the Board of Governors (BoG), 
including men and women, subject to availability at the day of the interviews. 


The complete sample achieved can be found in Table 5 below and the full list of survey 
respondents and interviewees can be found in Appendix C. 


Table 5. Achieved sample 


Number of Respondents’ profile 


respondents 

Basic school 10 Seven Head Teachers, two Deputy Head Teachers and one 
survey and Director of Studies (DoS) 
wale 70 percent male and 30 percent female 
Case study 10 Six respondents with leadership and management 
interviews responsibilities (Deputy HT, DoS, assistant DoS, school 
Group 1 bursar), three members of the BoG and one Head of 

Department 

70 percent male and 30 percent female 
Case study 10 Three respondents with leadership and management 
interviews responsibilities (Head Teacher and Deputy HT), two 
Group 2 members of the BoG, three Heads of Department and two 

teachers 

60 percent male and 40 percent female 
Key a Four interviews conducted with PEAS: three staff members 
stakeholder and one peer school leader mentor. Two female and two 
interviews male. 


Three interviews conducted with DES: Two national-level and 


one district-level official. Two female and one male. 


3.5 Research Limitations 
We note the following limitations to this evaluation: 


e Lack of control group: due to practical considerations around sampling and costs, PEAS and 
NFER chose to forego the use of a comparison group. To address the challenge that this 


imposes around causal attribution we chose a comparative case study approach. This approach 


allows to compare the impact of the intervention across treatment schools with different 
contexts. 
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e COVID-19 repercussions to WMS as an outcome measure: To try to reflect the changes that 
would have resulted from the |&I pilot rather than the school closures resulting from COVID-19, 
we asked school leaders to reflect upon the status of school management practices prior to the 
outbreak of COVID-19. Therefore, changes to WMS scores may represent changes to 
management approaches or include an element of recall bias resulting from recalling events a 
year prior. Although WMS scores changed from baseline to endline, we found little correlation 
between the two measurements. For these reasons, the absolute WMS scores should be treated 
with caution as they reflect the aggregate state of management practices in schools at two 
points in time, rather than changes that can be directly attributed to the I&I pilot. Nevertheless, 
we triangulated WMS data with qualitative findings from case study and stakeholder interviews 
in order to draw conclusions on changes to management practices and to elucidate the links 
between I&l and the observed changes in WMS scores. 


e High rates of turnover in the head teacher position: We encountered a high rate of turnover 
amongst our respondent population of head teachers. As a result, some respondents did not 
have sufficient information about the I&I pilot, as they had only been in post for less than two 
years at the time of endline data collection. This affected the data quality as head teachers could 
only share perspectives based on their limited participation in the pilot. In order to mitigate 
against this, the evaluation team, in collaboration with PEAS, instead identified deputy head 
teachers who had been in post through the duration of the pilot. 


e Change in data collection team: Due to the extended time period between baseline and 
endline data collection, we employed a different group enumerators for data collection at 
endline. This has the potential to introduce a different bias in the way in which the WMS is 
scored at endline. However, in order to mitigate the risk of bias, we conducted rigorous training 
at endline and introduced a quality assurance process to further validate WMS scores. This 
included rescoring a random selection of WMS scores from both baseline and endline, using 
audio recordings and transcripts. We found only minor discrepancies in the WMS sub-areas 
selected for the spot check, and therefore felt that this limitation has been sufficiently mitigated. 
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4 Findings 


This section presents a summary of the results and analysis from the endline evaluation. The 
analysis comprises of and triangulates information from the survey, case study and key 
stakeholder interviews. The section outlines results from the WMS (Section 4.1), the impact of the 
pilot in school leadership and management and the mechanisms through which change happened 
(Section 4.2), the enablers and barriers to improvement in school leadership and management 
(Section 4.3), the impact of the pilot on other areas of school operations (Section 4.4 and finally, 
the perceived value of I&I for its stakeholders (Section 4.5). 


4.1 What impact did the I&I pilot have on school leadership and 
management practice? 


4.1.1. What can the WMS tell us about leadership and management in pilot 
schools? 


This section presents the findings from the WMS conducted in all ten pilot schools before and after 
they received the support from I&l. Data collection took place in February/March 2019 (baseline) 
and February 2021 (endline), the latter reflecting on management practices before COVID-19 
related school closures. The adapted WMS allowed us to assess school management in five areas 
and 12 sub-areas (See Table 6). 


Table 6. Description of adapted WMS areas 


Area Sub-area Description 
Operations 1. Standardisation of Tests how well materials and practices are 
instructional standardised and aligned in order move students 
processes through learning pathways over time 
2. Personalisation of | Tests for flexibility in teaching methods and student 
instruction and involvement ensuring all individuals can master the 
learning learning objectives 
3. Data-driven Tests if the school uses assessment data to verify 
planning learning outcomes at critical stages, making data easily 
available and adapting student strategies accordingly 
4. Adopting Tests if the school incorporates teaching best practices 
education best and how these resources into the classroom 
practices 
Monitoring 5. Continuous Tests if school has processes for continuous 
improvement improvement and staff attitudes towards change 
6. Performance Tests if school performance is measured, reviewed, 
tracking, review discussed and followed up with teachers with the right 
and dialogue frequency and quality 
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Target- 7. Balance of Tests if the school has meaningful and well-planned 
setting targets/metrics targets -at the school, teacher and student level- linked 
to student outcomes 


8. Target Stretch Tests if school sets targets with the appropriate level of 
difficulty to achieve 


9. Budgeting Tests if the school has processes for planning, 
monitoring and adjusting their budgets 


Leadership 10. Leadership vision — Tests if school leaders have an understanding of the 
broader set of challenges that the school, system and 
key actors face and the right mind-set to address them 


11. Clearly defined Tests if school leaders are accountable for delivery of 
accountability for | student outcomes 
school leaders 


People 12. People Tests if school identifies and targets teaching needs 

management management and capacity gaps, how they find, motivate and retain 
the right teachers, and how they handle bad 
performance 


Enumerators provided scores from 1 to 5 based on the head teacher's responses around practices 
in each sub-area. In general a score of 1 can be interpreted as ‘practically no structured 
management practices implemented’ and a score of 5 represents ‘well-defined strong practices in 
place which can be seen as best practice’. The specific scoring criteria for each sub-area can be 
found in the WMS rubric in Appendix E. 


At endline, the strongest management practices demonstrated in pilot schools were related 
to the areas of monitoring and people management 


At endline, monitoring and people management were the two areas where pilot schools had more 
established management practices before COVID-19 induced school closures, both with an 
average WMS score of 3.9 out of 5. 


The monitoring area assesses processes to track and disseminate school performance, continuous 
improvement, mechanisms to learn from problems and staff attitudes towards change. Intervention 
schools scores ranged between 3 and 5, with one school scoring 3. The people management area 
comprises of processes to find, recruit, motivate and asses the performance teachers, as well as 
consequences for bad performance and rewards for good performance. School scores in this area 
ranged from 2 to 5, with one school scoring 2. This suggests that the majority of pilot schools had 
formal management processes in these two areas, in some cases consisting of reactive practices 
and in other cases consisting on a more proactive approach to managing the school. 


In terms of sub-areas, pilot schools had the highest average WMS scores in standardisation of 
instructional processes (4.4), continuous improvement (4.4), budgeting (4.2) and people 
management (3.9). 
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Compared with WMS baseline measures, pilot schools had the largest improvements in 
target setting and people management practices and the least improvement in leadership. 


The average WMS score across all schools at endline was 3.7, representing an improvement from 
the baseline WMS average score of 2.3 (See Figure 2). There was also greater variation in scores 
across schools at endline since the standard deviation was 0.5 compared with 0.2 at baseline. 


Figure 2. Change in average WMS scores in I&l pilot schools 
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Note: baseline scores correspond to management practices in pilot schools in March 2019, endline scores correspond to 
management practices in March 2020 (prior to COVID-19 school closures). Endline data was collected in February 2021. 
When comparing the performance of schools in each WMS area in before and after I&l, we found 
the largest improvements in people management and target setting practices. In general, the 
change in scores suggest that pilot schools went from having some informal (and in some cases 
unstructured) management practices to more formal practices across all WMS areas. 


Although schools demonstrated an improvement in target-setting practices, this continued to be the 
area with the lowest score. Improvement was mainly driven by enhanced budgeting practices, 
which was one of the twelve WMS sub-areas with the largest improvement from baseline to 
endline (going from 7" to 3" highest score). On the other hand, the sub-areas of balance of 
targets/metrics and target stretch continued to be the weakest sub-areas at endline. These sub- 
areas assess whether the school has well-planned targets at the school, teacher and student level 
and whether they are of the appropriate level of difficulty to achieve. Given the low starting point for 
these two sub-areas, the modest changes suggest that the process of improvement centred 
around the need to put in place basic target setting processes that in some cases, were not yet in 
place — rather than focused on improving existing processes. 


The area with the smallest average improvement was leadership, falling from second highest score 
at baseline to second lowest score at endline. Leadership comprises the process to set and 
communicate the school vision, the school leaders’ understanding of the vision and the extent to 


Evaluation of the PEAS-DES Inspect and Improve Project Endline Report IG 


“NFER 


International 


which the school leader is accountable for school outcomes. The lack of change given the high 
ranking at baseline suggests that this is both an area in which school leaders demonstrated 
existing processes (for instance, school visions were already in place at baseline) as well as an 
area in which school leaders have less autonomy, as improving accountability requires changes to 
wider education governance structures and processes beyond the school leader’s control. 


4.1.2 What did the beneficiaries and stakeholders perceive as the impact of I&l in 
school leadership and management? 


The WMS scores provided a means to identify comparative case studies. We identified two cases 
with a WMS change below average and two with a WMS change above average to represent 
differentiated impacts resulting from the |&! pilot. The comparative case study approach allowed us 
to explore similarities and differences between schools that have moved up or down the ranking of 
WMS scores from baseline to endline. This in turn allowed us to identify patterns of the perceived 
impact of the I&l project in different school contexts. 


In the following section, we reflect on the WMS scores presented in Section 4.1 and triangulate 
these findings with data from the basic school survey, case studies and stakeholder interviews. 
This allows us elucidate not only the link between the I&I pilot and the observed changes in WMS 
scores, but also how the status of management practices, as measured by the WMS, influences 
the way that school leaders and school staff perceived the impact of I&l (see Box 2). 


1&l helped school leaders understand the scope of their role and develop skills to 
successfully monitor teaching and learning 


One of the main objectives of the I&l project is to stimulate long-term improvement in the quality of 
leadership and management by supporting school leaders to implement strategies to improve their 
schools. These strategies respond to the school’s areas for improvement identified in the initial 
inspection that is carried out by PEAS and DES. A key requisite for achieving this objective is that 
school leaders assume an active role in tracking the progress made on the improvement 
objectives, which implies closer supervision of teachers and students and stronger internal 
monitoring tools and processes. It was widely mentioned across respondents that with the 
introduction of |&l there has been an observable change in the administrators’ approach and 
frequency of school and class supervision. One teacher mentioned since the start of |&l, they 
noticed ‘more seriousness in relation to conducting school activities and demanding better 
performance’. 


Overall, school staff respondents viewed these changes in supervision and monitoring positively; 
as one interviewee pointed out: 


Before I&l, administrators were not so much into monitoring teachers or having good 
interaction with them. There was some distance and teachers had a negative attitude 
towards the administrators. Now the school leader shows teachers the right thing to do, at 
the right time. Teachers understand that administrators are not here to penalize but to 
shape so that they come up with good results. (School staff) 


Respondents largely agreed that !&I activities helped not only to develop the supervision skills of 
head teachers but also shed light on the role of teachers in the school’s improvement process. The 
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direct impact of the pilot on monitoring processes in two levels (administration and teaching) 
appears to be one of the main catalysts of further changes in case study schools. School leaders 
enhanced (or in some cases, developed) their internal supervision tools and skills to keep track of 
attendance and teaching practice in the classroom. 


Box 2. The impact of |&l on leadership and management was perceived differently 
according to how established management practices are in the school 


Whilst there was a consensus across all schools on the positive changes in leadership and 
management, the type of impact perceived differed slightly according to how developed the 
management practices were in each school. Respondents from schools with lower WMS 
scores emphasised the impact of |&l on improving foundational administrative processes, 
such as more lesson observations, regular meetings between administrators and staff, 
record keeping, and identifying weak areas in the school. Respondents from schools with 
higher WMS scores (e.g. schools with more established management practices) 
emphasised changes at a more strategic level, related to planning, target-setting and 
accountability. This slight distinction in schools’ emphasis when talking about the impact of 
I&l points to the personalised nature of the support given by the project and how it 
successfully adjusted to individual school needs. 


School leaders were able to improve target setting, which resulted in better resource 
prioritisation, accountability and transparency 


There was consensus amongst respondents that improved transparency and accountability were 
key changes brought by I&l, particularly with regards to administrators sharing information about 
school finances to the staff. Furthermore, progress in this dimension was two-way, as more regular 
teacher appraisals also improved accountability from teachers to school leaders. School leaders 
felt that improved accountability also ensured teachers and staff were clear on what was expected 
of them, and allowed head teachers to delegate some responsibilities. 


Financial accountability was possible due to school leaders building financial management skills. 
School leaders reported that |&I helped them to prioritise areas of the budget in order to work 
towards the objectives set out in the SIP (See Box 1 in Section 2.2). A key part of progress in this 
area is that school leaders and PEAS work together in setting up a SIP that takes into account the 
school resources so that its implementation is realistic. As one stakeholder put it: 


PEAS taught administrators to make a SIP, having in mind the limited resources that the 
school has, not going beyond these resources. The SIP enabled them to economise and 
yet move forward. (School leader) 


Better resource allocation was also a result of better target setting. Respondents largely agreed 
that I&l, and particularly the SIP, helped them set appropriate targets, in line with the identified 
weaknesses and gaps of the school. 
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1&l helped school leaders enhance people management skills, improving communication 
and working relationships between administrators and staff 


Some of the dimensions of the people management area in the WMS correspond to highly 
centralised practices such as recruiting, hiring or firing teachers, in which school leaders have little 
autonomy, particularly when managing staff on government payroll. In Uganda, a centralised 
statutory body, the Education Service Commission, is responsible for recruitment, termination, 
promotion, performance management of teaching and non-teaching staff in secondary schools. 
Where staffing gaps exist, school leaders collaborate with Parent Teacher Associations (PTAs) to 
recruit and manage additional personnel. Individual school leaders’ autonomy over people 
management is limited to to this category of school personnel. Nevertheless, I&! helped school 
leaders develop skills in areas in which they have more autonomy and therefore more room for 
improvement, such as identifying good and bad performance, keeping teachers motivated, 
rewarding good performers and dealing with underperformers. 


1&1 was instrumental in improving a dimension of people management in which communication and 
working relationships between school leaders and school staff, school leaders and governing 
bodies (BoG, PTA, SMC), and between teachers and learners. Setting and communicating targets 
and involving school staff in decision making processes built a sense of ‘team work’ in schools and 
increased staff morale. As one head teacher put it: ‘/&/ brought an unexpected change in the 
laissez-faire attitudes and mind-set of teachers, which has made the work of the school leader 
easier.’ 


As school leaders learnt how to communicate assertively to teachers and identify weaknesses and 
opportunities for improvements after classroom observations, teachers changed their perception of 
inspection and lesson observation from a punitive activity to a more open conversation based on 
constructive feedback. In some cases, teachers expressed greater interest in being observed as 
they felt more confident in their improvement of teaching practices based on lesson planning. One 
head teacher mentioned that: 


1&l drew management closer to the teachers. After the inspection we had one-to-one 
discussions to tell them what they did well and what they needed to improve. Teachers 
started wanting to be seen. This has made appraisal process easier. (School leader) 


The enhanced supervision skills of the school leader combined with an open-door and interactive 
communication style had a ripple effect on the attitudes and mind-set of teachers, who realised the 
importance of their job and felt a valued part of the school community, and in turn to students, who 
perceived the increased commitment and effort that teachers were exerting in the lessons. 


Thanks to counselling, teachers understood that focusing in things like absenteeism is a 
matter of national standards and policies. It is something that happens in other places. It is 
not something of favouritism or discrimination. (School leader) 


Furthermore, I&l has bridged the gap between schools and DES inspectors. One DES stakeholder 
elaborated on this: 


There is a clear follow up process and a chance for the schools to plan with the inspectors. 
There is harmony in the collaboration. We are no longer seen as auditors. (DES official) 


Evaluation of the PEAS-DES Inspect and Improve Project Endline Report 18 


"NFER 


International 


4.2 In what ways did I&l contribute to the changes in school leadership 
and management? 


In this section, we explore the different support mechanisms used by |&! and identify the ways in 
which respondents perceived the contribution of the pilot to impact explored above. 


The SIP emerged as the most effective form of support and became a management tool for 
school leaders 


Pilot schools reported receiving different types of support aimed at advancing school improvement. 
These included: 


e PEAS and DES inspection visits 

e SIP development workshop with head teachers 

e PEAS inspection team monitoring visits 

e PEAS head teacher peer mentoring visits 

e support calls from PEAS inspection team and peer head teachers, and 
e further head teacher training in some cases 


Results from the basic school survey indicate that pilot schools received on average six calls from 
peer head teachers and six calls from PEAS staff. They also reported on average three visits from 
peer head teachers and five visits from PEAS staff. All schools received at least one session of 
head teacher training with the SIP workshop, and three schools reported receiving additional 
training sessions. Furthermore, regional DES staff also received training support from PEAS during 
the pilot. 


In the basic school survey, we asked school leaders how effective they thought various support 
mechanisms were in helping them improve leadership and management in the school. The SIP 
emerged as the most effective strategy, followed by PEAS staff and peer head teacher support. 


School leaders reported that they were able to use it as a management tool to track progress 
against objectives set as a result of initial school inspections and were able to effectively 
operationalise and action on change strategies. SIPs were regarded as progressive rather than 
stationary, which in turn increased school leaders’ ownership of supervision and monitoring 
processes at the school. 


The SIP also contributed to change the perception that school leaders and teachers had of the 
inspection process. Prior to the I&I pilot, inspections were considered a one-off external exercise, 
somewhat disconnected from schools operations. Data from the case study interviews and 
stakeholder interviews revealed that inspections are now recognised as part of a continuous 
process that require follow-up and can be organically embedded into the head teachers’ 
management practices. School leaders therefore have come to see inspections not as an end in 
itself but as the starting point of self-directed monitoring activities in the school. 
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The SIP was widely linked to better target setting and therefore prioritisation of resources, 
reconfirming the findings from the internally-conducted midline evaluation report, in which school 
leaders mentioned that the SIP enabled them to focus on their strategic objectives. 


School leaders perceived support and monitoring visits as key to school improvement and 
believed that schools can benefit from more frequent visits 


There was consensus across case study respondents on the importance of regular support visits to 
follow up on SIP progress and to build a comprehensive picture of the school’s environment, 
including personal characters, leadership/management styles and ways of communication. 


School leaders pointed to the effectiveness of visits in two different ways: while some more highly 
valued the supervision and monitoring approach by PEAS inspection team, others more highly 
valued PEAS peer head teacher support as a means to exchange ideas and to ‘learn by example’. 


Unsurprisingly, along with the initial inspections which involve DES officials’ per diems, 
stakeholders confirmed that monitoring visits were a more resource-intensive type of support as 
they require more planning, human resources and mobilisation costs. Nevertheless, they also 
recognised that this is the most important activity for schools: 


Support and monitoring visits required a lot of time and resources. Because you have to 
interrogate data and see evidence of improvement and sometimes you have to support and 
make further recommendations. But this is the most important for the school. This was the 
missing link of the DES inspections. (PEAS staff) 


We found that schools that reported a greater number of visits tended to have higher WMS scores. 
This positive association (see Figure 3)? supports the qualitative findings from case study and 
stakeholder interviews about the effectiveness of the follow-up visits in improving leadership and 
management in schools. 


Figure 3. Relationship between support visits and WMS endline score 


The number of self-reported support visits is positively 
associated with WMS scores 
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2 The Pearson correlation coefficient is 0.54. The self-reported number of visits is the sum of PEAS staff and 
peer head teacher support visits. 


Evaluation of the PEAS-DES Inspect and Improve Project Endline Report 21 


“NFER 


International 


School leader training was the only support mechanism that received one negative rating in the 
school survey, and on average was not perceived as effective as other support strategies. For two 
thirds of schools, there was no additional training other than the SIP workshop, while the need for 
training in foundational management skills emerged across all schools in the case studies. 


4.3. What were the enablers and barriers to improvement in school 
leadership and management? 


This section summarises findings from the case study and stakeholder interviews on the perceived 
drivers and barriers to change in schools. 


Strong working relationships and individual motivation underpin the capacity to adopt 
change in schools 


Strong working relationships were frequently mentioned by the majority of case study respondents 
as a key driver of change not only within the school, but also with external stakeholders such as 
governing bodies (BoG, PTA and district officers), parents and the surrounding community. School 
leaders acknowledged the importance of key stakeholders to engage with school matters and 
demand better school performance. Case study schools with higher WMS scores reported higher 
levels of collaboration with BoG, PTA and parents. One respondent associated the state of working 
relationships with the school’s capacity to adopt change: 


Some school leaders are involved with small struggles with the BoG, PTA or teaching staff. 
If they feel they are well settled in the school, they can focus on realising change. (PEAS 
staff) 


School leaders’ and teachers’ individual motivation and willingness to change emerged as a key 
enabler of change, as well as transparent and open leadership styles. This point was supported by 
responses from schools with lower WMS scores which identified leadership styles that bring 
divisions and abuse of power as a barrier to improvements, and reported more challenges with 
staff resistance to change and negative attitudes. 


Government schools trust the image of PEAS and DES as capacity builders and 
management experts 


Results from key stakeholder interviews revealed that the credibility of the 1&l partnership 
facilitated change. Stakeholders reported that the willingness to adopt changes in leadership and 
management practice can be partially attributed to head teachers’ recognition of DES as a duty 
bearer in school improvement. Additionally, respondents indicated that PEAS Uganda schools are 
perceived as high performing schools among head teachers in pilot schools. The positive 
perceptions towards PEAS Uganda schools may influence head teachers’ willingness to participate 
in the pilot activities and in turn drive the changes in individual practice. 


Regular head teacher transfers and changes in staff hinders the continuity in leadership 
and management reform processes 


The frequent turnover of head teachers in schools was largely mentioned across interviewees as 
one of the biggest threats to school improvement, as newly transferred head teachers may differ in 
terms of interpersonal skills and attitudes, training, experience and leadership style. For example, 
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within the pilot there was a change in school leader in four out of ten pilot schools in the duration of 
the pilot. 


Interviewees also identified changes in school staff as another barrier. One respondent mentioned 
that government teachers are transferred but not replaced, leading to a shortage of trained 
teachers in the school. In one pilot school, this has become a pressing issue that required the head 
teacher and BoG to develop a policy that defined a minimum service period of five years as a 
precondition before government transfer would be considered. 


Lack of autonomy over people management and limited engagement with governing bodies 
emerged as key barriers to improvement 


Gaps within the public sector performance management system also emerged as a barrier that 
inhibits improvement in school leadership and management. Data from the stakeholder and case 
study interviews highlighted structural issues associated with accountability in the workforce. For 
instance, head teachers have limited authority or ownership in decision-making related to 
recruitment, hiring and firing teachers employed by the Teachers’ Service Commission. As a 
consequence, where weak performance cannot be overcome, school leaders are faced with limited 
formal processes to address this. 


As previously mentioned, while strong working relationships acted as an enabler of change in 
some schools, poor working relationships and communication across governance structures were 
mentioned in schools with lower WMS as a barrier to improvement. Interviewees alluded 
particularly to mutual mistrust, lack of transparency, and disengagement of BoG and PTA 
members and parents. Generally, BoGs and PTAs varied widely in terms of how active they were; 
some did not necessarily meet during school closure and in some cases, BoG mandates had 
expired. 


Key challenges to teaching and learning changed from baseline to endline, mainly due to 
the effect of COVID-19 induced disruptions 


We asked head teachers at baseline and endline what were the key challenges to teaching and 
learning faced by the school. Figure 4 shows the challenges that were reported by more than one 
school at endline, which allude to the period since schools reopened (October 2020) until data 
collection in February 2021. The top three in order of prevalence are: ‘school budget is 
inadequate’, ‘students are absent’ and ‘students are dropping out of school’. 
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Figure 4. Key challenges to teaching and learning reported by pilot schools at endline 


60 percent of schools mentioned school budget constraints as a key 
challenge to teaching and learning 


Number of schools reporting the challenge 


School budget is inadequate 
Students are absent 

Students are dropping out of school 
Inadequate school insfrastructure 


Inadequate teaching materials 


Local attitudes affecting 
girls’ access to education 


Teachers are not motivated 


Teachers are absent 


In the basic school survey, 60 percent of pilot schools mentioned budget constraints as a key 
challenge, which was also widely mentioned across the case studies interviews. Three main 
reasons for funding shortfalls emerged in the case studies. Firstly, parents whose income has been 
affected by COVID-19 were unable to pay school fees. Secondly, as enrolment fell in schools’, 
per-pupil USE transfers also decreased. And lastly, reduced income was accompanied by an 
increase in operational costs due to public health compliance requirements. 


These findings are validated by the notable change in results from the schools survey at baseline 
and at endline. At baseline, only one in ten schools reported school level funding shortfall, 
compared to six in ten schools at endline. 


A key consequence of this is exemplified in schools’ reported difficulties in paying salaries for 
teachers who were not employed by the government. This category of teachers, recruited to cover 
critical gaps in public service staffing at school level, were compelled to take pay cuts during the 
crisis. Respondents from case studies and stakeholder interviews attributed the decline in teacher 
morale and motivation to these changes in working conditions. Interviewees also mentioned that 
insufficient financial resources undermined SIP implementation and school infrastructure 
development. 


Although local attitudes affecting girls’ access to education was reported as a challenge by only 
two schools in the survey, it was largely mentioned in case study interviews that girls have been 
affected more than boys by COVID-19, as some of them have married and/or have become 
pregnant during school closures. According to data form the survey, in four pilot schools the 


3 According to data from the survey, the average school enrolment in pilot schools went from 523 students at 
baseline (school year 2018) to 327 students post COVID closures (Feb 2021). 
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proportion of girls enrolled in February 2021 fell between 10 and 27 percentage points as 
compared to baseline measures. 


COVID-19 pandemic containment measures accentuated existing challenges to school 
leadership and management and threatened progress in schools 


There was consensus amongst interviewees that the prolonged school closure had a significant 
impact on core areas of school leadership and management. Schools deferred implementation 
activities on their SIPs, while PEAS Uganda substituted provision of monitoring support visits with 
follow up phone calls until the partial reopening of schools commenced. The case studies showed 
that schools with smaller improvements in WMS scores reported more examples of financial 
constraints than schools with larger change in WMS scores. 


We asked in the survey what challenges to school leadership and management have emerged or 
were made worse by COVID-19 disruptions. Figure 5 presents a summary of challenges reported 
by more than one school. 


Figure 5. Self-reported challenges to school leadership created or accentuated by COVID-19 
disruptions 


More than half of schools mentioned lack of collaboration with local 
entities as a key challenge to school leadership 


Number of schools reporting the challenge 


Lack of collaboration with local entities (e.g. PTA’s) 
Tighter reporting to authorities 


Focus on short-term planning 


Changes in teacher’s attitudes towards school 
leadership 


Deprioritised areas of curriculum 


Little autonomy over decision making 


Insufficient capacity to implement re-opening safety 
guidelines 


Lack of collaboration with local entities was the top challenge reported by 60 percent of schools. 
This was corroborated in the case studies where interviewees largely mentioned lack of 
commitment and disengagement of the BoG and PTA as a barrier to school improvement. Half of 
the schools reported tighter reporting to authorities as a challenge brought by COVID-19. Although 
this did not come up explicitly in interviews, school leaders did mention the difficulty in complying 
with the government Standard Operating Procedures (SOPs) instituted by Uganda Ministry of 
Health for school reopening. 


Additionally, interviewees reported deterioration in student wellbeing, evidenced by increased 
absenteeism and psychological distress. Respondents highlighted examples of students not 
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reporting as required, missing substantial periods of the school term and teachers’ fears of 
contracting COVID-19. Some interviewees mentioned that one of the causes of student 
absenteeism and lack of motivation was the interruption of co-curricular activities to comply with 
reopening safety guidelines, particularly sports and outdoors activities. 


COVID-19 disruptions presented challenges to keep school leaders engaged with 1&l 
activities 

Some of the school leaders mentioned that they received calls from PEAS during closures to 
advise and check progress with the SOPs and to ‘give them encouragement. Additionally, the 
frequency of vital stakeholder engagements such as BoG meetings was reported to have reduced 
significantly during this period. A project staff member elaborated on the challenges of sustaining 
school leaders’ engagement during the period of extended school closure, mentioning that: 


Some head teachers switched their phones off, they went off track. PEAS tried to call them 
but they were on holiday. (PEAS staff) 


Forty percent of head teachers in pilot schools indicated that the skills developed with |&| 
moderately helped them to mitigate the impact of COVID-19, while 60 percent felt they ‘somewhat’ 
or ‘not at all’ acquired the management skills to handle this situation. One school leader recognised 
that the I&l emphasis on involving governing bodies, parents and community in the school was 
instrumental to help with the challenges of COVID-19. They mentioned that: 


When COVID came, PEAS encouraged them to call the parents and ask them which type 
of school they wanted. The head teacher was also able to ask them to make contributions 
towards the purchase of the items that were used for SOPs. (School! leader) 


Project staff cited having a functional SIP as a key preparedness strategy that enabled pilot 
schools to mitigate the impact of disruptions. One PEAS staff member observed: 


Having a SIP in place helped them to know where they are, pick them up. There was 
structure in place to manage the transition during school reopening. (PEAS staff) 


4.4 What other areas of the school’s operations did the I&I pilot affect? 


This section presents findings from the survey and the reflections from interviews on what school 
leaders, school staff and stakeholders perceived as the extended impact of I&! on other areas of 
school beyond leadership and management. 


School leaders perceived that improvements in leadership and management, as a result of 
1&1, have positively affected other areas of schools’ operations 


Evidence from case studies suggest that the improvements in /eadership and management have 
facilitated change in other areas of the school, including quality of teaching and learning. Most of 
these improvements derived from the impact outlined in Section 4.2, related to enhanced 
supervision, planning and communication. One respondent in the case study interviews mentioned: 


Changes in school leadership and management practice have contributed to other changes 
because it has helped bridge the communication gap between the school staff and admin. 
(School leader) 
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Overall, schools that reported higher improvements in other areas of school operations also had 
higher baseline WMS scores. This suggests that the starting point of schools in terms of 
management practices is associated with their improvement path in other areas of the school. 


In the absence of final I&l inspections due to school closures, we asked school leaders to score the 
extent of the changes observed in seven areas of school operations before school closures on a 
scale from 1 to 5, where 1 meant ‘no observable changes’ and 5 ‘improvement objectives were 
met’. These areas are the same as those assessed in the DES inspection tool and in turn inform 
the SIPs: teacher attendance, learner attendance, learner achievement, teaching quality, leaner 
behaviour, learner safety, and school leadership and governance. Figure 6 presents a summary of 
self-reported scores of the progress in other areas of school operations. 


Figure 6. Self-reported progress in other areas of school operations, as outlined in DES 
inspection tool and the SIP 


School leaders indicated that they were closer to meeting their 
improvement objectives in teaching quality 


Teaching quality a 
Teacher attendance SEE -° 
Learner safety = 
School leadership and ® 
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Learner behaviour 3.6 
Learner attendance 3.6 
Learner achievement a Y 
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1: No observable changes, 5: Improvement objectives were met 


Results show that all schools perceived positive changes in all areas of the SIP. Specifically, 
improvements in teaching quality and teaching attendance emerged as the most frequently 
mentioned observed change across all data sources while the lowest reported improvement was 
on learner achievement. 


As these are self-reported changes, they are influenced by individual staff and school contexts. 
Although all case study schools recognised the impact of |&I beyond school leadership and 
management, we found that school leaders and staff noticed and valued different types of changes 
to differing degrees, according to how well established their management practices were. Schools 
with lower WMS scores tended to focus their responses about the drivers of I&l value on back-end 
aspects such as ways of working and lines of accountability in the school, relationships with 
administrators, decision making and problem solving processes. On the other hand, responses 
from schools with higher WMS were more focused around front-end impact, for example teacher 
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motivation, lesson planning, teaching practices, non-corporal disciplinary methods, and better 
communication between teachers and students. 


Pilot schools reported improvements in teaching quality including pedagogy, curriculum 
planning and coverage 


Respondents reported noticeable improvements in the quality of pedagogical practice, lesson 
preparation, development of schemes of work and syllabus coverage. This includes adoption of 
interactive approaches and completion of syllabus. Respondents attributed these changes to 
increased supervision support by school leaders. As teachers were monitored more frequently and 
received timely feedback, they realised the importance of planning lessons, covering the syllabus 
and delivering quality teaching in general. Teachers also reported changes in their peers’ attitude 
towards classroom observation: 


Now teachers are positive about classroom observation, they don’t run away. Now teachers 
come prepared and even ask the administration to come and observe their lessons. 
(School staff) 


There was consensus amongst respondents that I&I contributed to improving relationships 
between teachers and students. In terms of pedagogy, some teachers pivoted to more 
participatory and practical learning, by planning and using teaching materials, and asking students 
questions to check their understanding. Respondents from multiple schools illustrated this point: 


Before the inspection, lessons were focused on theory, now teachers tried to make them 
more practical. We even had student trips in geography and agriculture. (School staff) 


Now we have schemes of work, lesson plans, record of syllabus coverage, and effective 
use of the library. Teachers are now in a position to ask learners to use the library materials 
in order to get the content that is required. (School staff) 


School leaders observed that student attendance and teacher presence in school and in 
class improved as a result of enhanced internal monitoring 


Increased teacher attendance and student attendance emerged as the second most observed 
change according to results from the basic school survey. Case study interviewees attributed these 
changes to increased monitoring support processes introduced by school leaders in the !&I pilot. 
Evidence from case studies indicate that school leaders institutionalised the use of monitoring tools 
to document teacher attendance and punctuality for curriculum activities. One respondent 
mentioned: 


The teachers became committed in their work, serious and active [...] They start class on 
time and teach the correct subject at the correct time. According to the monitoring forms, 
the lesson attendance by teachers went up to 75 percent from around 20 percent. (School 
management stakeholder) 


In addition to the monitoring tools, school leaders adopted what is described as ‘learning walks’, as 
one school leader elaborated on: 
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One thing that we have learnt from the 1&/ project is that school administration is done best 
from outside (school facilities), but not sitting in office. (School leader) 


The basic school survey’ and internally-conducted midline evaluation findings suggest there was 
an upward trend in school enrolment in I&I pilot schools before COVID-19 closures. School leaders 
from two case study schools mentioned that the I&l strategies to increase parental involvement 
contributed to this change in enrolment, since parents became more interested and motivated 
about school matters. In one case, motivated by the improvements evidenced in the schools, some 
parents advertised the school in their communities. 


Pilot schools adopted more strategies to enhance learner safety and well-being 


I&I pilot schools also reported increased investment and attention to core areas of learner safety 
within their SIPs. Respondents attributed these changes to improvements in prioritisation and 
target setting among school leaders. Schools prioritised different issues in the spectrum of learner 
safety according to their context (see Box 3), for example, some schools focused on reducing the 
use of physical punishment, while others focused on the enhancement of school facilities such as 
construction of school dormitories and fences. One case study respondent stated: 


We have learned how to handle a number of issues like the discipline of learners. Formerly 
we used to do a lot of corporal punishments but since this intervention came, we stopped 
that. (School staff) 


Box 3. Schools’ approach to student safety differed according to their pre-existent 
infrastructure gaps 


While general improvements in learner safety were reported across all schools, the comparative 
case studies revealed marked differences in specific changes adopted to enhance learner 
safety. Both schools with larger change in WMS scores mentioned reduced use of physical 
punishment as a key observable change. In contrast, schools with lower WMS scores focused 
their responses in physical infrastructure and facilities, particularly dormitories and fencing. This 
suggest that schools which face greater gaps in infrastructure are more likely to prioritise these 
issues over corporal punishment. 


One of the criteria of learner safety and well-being indicated in DES inspection tool is the provision 
of midday meals for students. As a result, the provision of school meals was prioritised in SIPs in 
some of the pilot schools. School leaders were able to mobilise parents to co-finance the school 
development fund to provide meals, in order to support student attendance. Both school-based and 
stakeholder respondents attributed increased learner attendance in these schools to successful roll 
out of the school meals initiative. One respondent mentioned: 


During the baseline, 5 percent of children in one school were receiving school meals. After 
prioritising this in the SIP, this number has gone up to 60 percent. (PEAS staff) 


4 According to data from the survey, the average school enrolment in pilot schools was 608 students in the 
school year 2019, compared with 523 students on average in the school year 2018. 
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4.5 What was the perceived value of I&I? 


School leaders and stakeholders both felt that improvements brought about by I&l justified 
the costs of PEAS-DES joint inspection process 


We asked school leaders from all pilot schools whether they felt that the time and money spent in 
I&l activities was justified by the changes to leadership and management and more broadly to 
teaching and learning. Responses were overall positive, 70 percent of school leaders felt that the 
investment was justified by changes in leadership and management and 100 percent of school 
leaders thought that the investment was justified by improvements in teaching and learning. With 
the case study interviews, we explored further the perceived added value of I&l in pilot schools. 
The scope of the perceived value of I&l by school leaders and school staff, presented in sections 
4.2 and 4.4, was wide and included enhanced teachers’ motivation and attitudes; capacity building 
for school leaders, teachers and BoG members; improved teacher and student attendance; greater 
engagement of governing bodies; greater delegation; improved working relationships, amongst 
others. 


We asked PEAS and DES stakeholders about the perceived cost of the I&! inspection approach, 
as compared with PEAS-only and DES-only alternatives. For this purpose, they recognised two 
components of the !&l inspection process: inspection visits and school improvement support. 
Regarding the inspection visits, stakeholders indicated that although PEAS-DES joint inspections 
were more expensive than PEAS-only inspections, they could have a cost advantage over DES- 
only inspections. As compared with PEAS-only inspections, DES officials’ per diems and limited 
availability to engage with activities represented higher operational costs and some logistical 
challenges. According to DES stakeholders, the cost of PEAS-DES joint inspections could be less 
than DES-only inspections as they had a geographical focus (i.e. involving schools within a 
catchment area) and engaged locally-based district inspectors and associate assessors. DES also 
reported that the joint inspection visits were more cost-effective since the PEAS and DES team 
were able to maximise their time by observing more lessons and covering more areas in the school 
visit, and it also allowed them to prepare the inspections report on site, which avoided delays in 
implementation. 


There was consensus amongst stakeholders that the key comparative advantage of the PEAS- 
DES joint inspection process, as compared to government inspections, is the focus on school 
improvement, as one interviewee pointed it out: 


I&l added value to the missing link in DES inspection practice from the time of inspection to 
putting into practice; translating inspection and results to improve performance. | think that 
is an area which we can leverage on, we need to document it better so that it can be 
replicated, so that in the new role out of the project that should come out very clearly. (DES 
Official) 
The school support component of the I&! approach would contribute to increasing costs as 
compared to the government inspections, as it involves further school visits by the inspection team 
and PEAS peer head teachers. In this regard, PEAS and DES have made efforts to find cost- 
effective solutions such as monitoring progress and providing support through phone calls and 
WhatsApp groups. Although in general |&l support mechanisms were well received by schools, 
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phone calls were not frequently mentioned in the case study interviews, suggesting that efforts are 
needed to ensure that schools can benefit effectively from other types of remote follow-up. 


PEAS and DES stakeholders largely agreed that the I&I inspection process demonstrated value for 
money and that the benefits of the joint approach outweighed the costs, which are seen as 
investments aiming at strengthening the MoES approach to inspections and, ultimately, long-term 
school improvement. The |&I pilot provided useful and timely feedback on how to improve the 
newly developed DES inspection tool (developed in 2019) and harmonised it with the improvement 
process. Both PEAS and DES benefited from bilateral capacity building as a result of training and 
hands-on experience in conducting joint inspections. By working alongside DES, PEAS gained 
school and wider national stakeholder buy-in, availed the expertise from government inspection 
process, and better understood policy priorities and needs with regards to school improvement. For 
DES, value for money was driven by the introduction of improvement plans, which closed an 
existing gap in DES inspection cycle that currently ends with handing out the inspection 
recommendations to schools. DES reported that one disadvantage to joint inspections might be 
that government schools would perceive an advantage held by PEAS schools, causing feelings of 
inequality or a negative reaction to PEAS involvement in school improvement. On this point, some 
school leaders suggested creating networks of knowledge-and-practice sharing with neighbouring 
government schools, which could also leverage the cost-effectiveness of geographic proximity. 
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5 Conclusions 


The aim of the evaluation was to support the I&I project to better understand what approaches can 
improve school quality, and how, if effective, these could be rolled out by MoES at a larger or 
national scale. Our conclusions summarise our findings, firstly on the effectiveness of the I&I pilot 
in improving the quality of leadership and management and the reasons why we believe the 
programme was effective. We then identify further observations from the experience of the pilot to 
generate lessons learned for further possible scaling of the project. 


5.1 Was I&l successful in improving the quality of leadership and 
management and why? 


To understand whether the !&I pilot contains merit for further scaling, we first consider whether the 
pilot has generated sufficient evidence of its effectiveness and impact and an understanding of the 
mechanisms (the project activities and means to trigger change) by which the intervention 
produces change. 


There is strong evidence that Il&l was successful in improving the quality of leadership and 
management in all 10 participating schools across all areas of management as defined by 
the WMS 


Findings from the WMS indicate that, at endline, schools demonstrated strengths in monitoring and 
people management practices. Furthermore, looking from baseline to endline, WMS scores 
demonstrate that schools experienced the greatest improvements in the areas of target setting and 
people management. 


These findings were further reaffirmed through our case study research, which found that 
improvements to financial management and accountability improved school leaders’ ability to set 
targets for budgets. School leaders also improved the means by which they communicated with 
school staff. 


Examining changes to people management, case study respondents reported positive 
improvements in the ways in which school leaders were able to manage school staff by playing a 
more active role in supervising teachers and students and improving internal monitoring tools and 
processes. School leaders played a greater role in driving school improvement processes in terms 
of setting and tracking progress against improvement objectives. School leaders also improved 
their communication and working relationships with school staff. This supported their ability to 
provide guidance and direction for improved practices — for example in the use of classroom 
observations as a tool to support improvements in teaching practices. Improvements in 
communications also supported stronger relationships between school leaders and BoGs, as well 
as with DES inspectors. 


There is also emerging evidence that improvements to school management have led to 
improvements to student and teacher attendance, teaching practices, and student safety 
and well-being 


Our evaluation did not seek to measure changes beyond the project’s intended outcome of 
improved school management. Each of these areas of impact are observed in the school 
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inspection tool and therefore for many schools, are the area of focus for SIPs. Final school 
inspections to examine observed changes to these impact areas were unable to be carried out as 
a result of the outbreak of COVID-19 and related school closures. 


However, at the school level, evaluation respondents reported perceived further changes in their 
schools as a result of the participation in |&l. These impacts include changes to teacher motivation, 
which respondents attributed to the stronger supervision and relationships between school leaders 
and teachers. Respondents across multiple pilot schools reported observing changes to teaching 
practice, including to lesson preparation and the development of schemes of work and syllabus 
coverage and in the quality of pedagogical practice. Respondents also further reported higher 
attendance rates for both teachers and students, as a result of stronger monitoring practices. 
Finally, respondents reported important improvements to learner safety, including a greater focus 
on improving school facilities (such as dormitories and fences) and reducing the use of corporal 
punishment. 


Evaluation respondents identified SIPs and school visits as the two most effective 
mechanisms from I&l to support improved school management 


When asked to reflect upon which activities of the I&I pilot were the most effective in supporting 
changes to school management, the SIP and school visits were rated the most strongly. 


Respondents reported that they found SIPs to be an effective management tool that allowed them 
to track progress over time. SIPs allowed school leaders to identify relevant challenges at the 
individual school level and to develop a personalised improvement plan tailored to the needs of 
their schools. The use of SIPs also encouraged a change in how perceptions of assessments 
(such as observations and inspections) changed from being viewed as a one-off punitive measure, 
towards a process to support continuous improvement. 


School visits from PEAS staff and peer mentors from PEAS schools also provided individualised 
attention and supervision from PEAS support staff, as well as the opportunity to ‘learn by example’ 
from peers. 


5.2 What can we learn from the I&l pilot on the potential for scale? 


Having examined and validated the effectiveness of the I&l pilot and the mechanisms for impact in 
the previous section, this section examines the lessons learned from the I&I pilot on the conditions 
and potential for scale. In addition to effectiveness, we consider three further criteria for scale: 


e relevance (alignment with needs and fit with culture and context) 

e implementation quality (implementation fidelity and adaptation, delivery capacity, and 
dosage required for effectiveness) 

e implementation feasibility (including cost and value) 


In this section, reflecting upon our findings, we share some of the lessons from the I&I pilot that 
may be important to consider in the scale-up phase of the I&! project. 


Evaluation of the PEAS-DES Inspect and Improve Project Endline Report ad 


“NFER 


International 


5.2.1. Relevance 


Engaging school leaders from the start -and with the collaboration of other school 
stakeholders- ensures that SIPs and I&l support are relevant to school needs. 


All respondents in the basic school survey agreed that I&l strategies were aligned to the needs of 
the school (60 percent ‘completely aligned’ and 40 percent ‘moderately aligned’). SIPs were 
carefully tailored to the needs of individual schools through collaboration with school leaders, which 
helped bridge the gap between school inspections as accountability measures to national 
education governance and inspections as part of the improvement process. The I&l approach was 
also most successful when it involved cooperation amongst school leaders and governing bodies 
and school staff in order to further tailor the improvement process. 


Culture and mind set are important factors that require shifting before increased school 
management capacities can result in school improvements 


Respondents mentioned ‘mind set’, or positive views and attitudes towards improvement, as an 
important factor for the pilot’s success. Several school leaders indicated that they were not 
accustomed to providing feedback on the inspections and therefore improvement process. Where 
school leaders held amenable mind sets or where I&l was successful in shifting their mind set 
towards one of ‘improvement’, schools were able to enact wider management improvements. 


This shift is additionally important in creating buy-in and greater sense of ‘team work’ from across 
school staff and management, which was further supported when teachers and BoG members 
were more involved in decision-making processes. I|&l was able to have an indirect effect on 
parents as they are encouraged to participate in school activities and feel more motivated when 
they observe improvements in the school. Nevertheless, there is still work to do on strengthening 
community engagement and sensitising the community about the benefits of education. 


1&l is a timely intervention that falls within articulated government priorities for education 
reform 


School leaders and project stakeholders were also positive about the timely alignment of 
interventions with the priorities of schools and of the education system in Uganda. The majority of 
key stakeholders reported that the PEAS approach to inspections and school improvement 
addresses key needs of the MoES, as articulated in the Uganda National ESSP. Furthermore, the 
implementation of 1&l in government schools coincided with the initiation of MoES-led efforts to 
revise DES’ existing school inspection approach. This reaffirms the relevance of the |&l model with 
the strategic direction of government of Uganda, as it put emphasis on school improvement and 
the subsequent monitoring that goes beyond the inspection visit. A key project stakeholder 
elaborated on the existing gap within the existing school inspection approach: 


It doesn't make sense to report that 50 percent teachers are not attending to lessons 
regularly, and come back next year and measure the same. [It’s] better to use some of the 
resources to support schools to take on improvement strategies. (Project stakeholder) 


Multi-stakeholder engagement, at both the school level and at the national level, will continue to be 
an important factor to maintain the relevance of the |&l model. This includes supporting school 
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leaders to engage with school governing bodies and teachers to ensure wider support and buy-in 
for improvement. Engagement at national level is required to ensure that school management 
continues to be a policy priority to support improved learning outcomes. 


5.2.2 Implementation Quality 


The partnership between PEAS and DES in co-creating and implementing the I&l model was 
crucial to the pilot’s successful adaptation and implementation of approaches to 
inspections and improvements in government schools 


The I&l pilot appears to have successfully merged PEAS’ model of improvement plan with DES’s 
newly developed inspection tool. As part of this process, the pilot was able to contribute feedback 
to help support the DES inspection tool. One important difference between the PEAS and the DES 
inspection processes was a greater focus on the PEAS process with working with school leaders to 
unpack key gaps and their drivers through the SIP, whereas school leaders in government schools 
were only accustomed to contributing high-level information to the inspection with limited feedback. 


During the !&! pilot implementation, PEAS had mechanisms in place to receive feedback from 
school leaders and incorporate learnings throughout the project cycle. They did this through 
internal tracking tools and weekly implementation meetings; some of them with the presence of 
DES officials. The partnership nature of the project that highlights the roles of PEAS and DES as 
co-creators and co-implementers is crucial to reinforce the school support network and generate 
ownership and commitment to the process. 


One year was sufficient to observe changes to school leaders’ management skills and for 
school leaders and staff to perceive wider changes within their schools 


The results of both the WMS and case studies confirm that within the pilot timeframes, the pilot 
was successfully able to generate observable changes to both attitudes and mind set, as well as 
practice, in intervention schools. This was particularly evident in school monitoring practices, such 
as standardised supervision processes and practices. 


Even with the set back of COVID-19 related school closures, there is some evidence that school 
leaders were able to apply and use learned skills to help address new challenges. Where school 
leaders were able to improve their financial management capacities, they reported that they were 
better able to manage and make decisions on prioritisation in school budgets. This allowed school 
leaders to better address individual school challenges; school leaders further reported that this 
management skill was particularly important in strengthening schools’ resilience against external 
shocks such as extended school closures. 


Although the evaluation was unable to observe or confirm wider impacts to teaching and learning, 
there was a strong perception by respondents of emerging evidence for these changes. This was 
at least evident in the greater knowledge of better teaching practices. In some cases, teachers 
were, for the first time, using a systematic approach to prepare lessons, schemes of work and 
teaching aids, as well as actively working towards curriculum coverage in all grades. As one 
teacher put it: 
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Some of us who are still growing in the profession can say that we appreciate their [PEAS] 
efforts because it improves our skills and experience. When you continuously tell me to 
scheme and lesson plan, it builds my own experience in teaching. (School staff) 


It is likely that it will take more time for these changes to result in concrete changes to learning 
outcomes and assessment results, as well as further attention to any barriers that may impede the 
translation of improved teaching practices to learning outcomes. 


High turnover of school leadership and low foundational management capacities are 
related; these factors challenge the extent to which I&l can support school improvements 


An important barrier to the ways in which school leaders were able to engage with the I&l pilot was 
time in their role and foundational leadership and management skills. As revealed by the basic 
school survey, 70 percent of school leaders believed that they had the capacities to implement 
changes as set out by SIPs, while 30 percent only believed that they ‘somewhat had the capacity’. 
Although the lack of experience or lack of confidence did not impede school leaders, and thus 
schools, from benefitting from the I&I pilot, it may limit the extent to which they were able to benefit 
as compared to peers who had more established roles and experience. 


While there is no shortcut to increase one’s experience, school leaders focused on developing 
more foundational skills, processes and infrastructure improvements during the course of the I&l 
pilot. This is in contrast to school leaders who were able to focus on larger, more strategic changes 
to relationships with school stakeholders and in accountability and transparency. This indicates 
that school leaders who may be starting with less experience may require more time in their roles 
and I&l support to realise the same impact. 


5.2.3 Implementation Feasibility 


1&l has the potential to impact a wider number of schools through indirect impact to 
neighbouring schools 


An important consideration for scaling is whether the number of participating intervention schools is 
enough to sustain wider changes. As this phase of I&l was a pilot, the number of intervention 
schools was limited. However, there is promising evidence that the pilot has the potential to have a 
wider impact beyond targeted intervention schools. 


The first factor is related to evidence of the strong reputation held by PEAS as a credible source of 
school management expertise. The second factor is the reported interest and willingness of 
participating school leaders to build on the experience of participating in 1&1 to help support other 
neighbouring schools. Although school leaders in government schools did not necessarily believe 
that they operated in similar contexts or faced similar challenges to PEAS schools, participation in 
the 1&l pilot appears to have provided them with the confidence to share their learnings to schools 
which face similar challenges. Box 4 below provides further details. 
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Box 4. Schools involved in the I&l pilot strongly believe that other schools in 
Uganda could benefit from scaling up the project 


We asked school leaders in what ways they think the support provided by I&l can help 
other schools. Their responses were extremely positive and what they perceive as 
benefits of replicating the project came from different angles, based on their own 
experience in the !&l pilot. 


Schools with higher WMS scores emphasised on the benefits that the |&l scale-up 
programme could bring to more schools through a sort of ‘expansion approach’ that 
provides training for head teachers and teachers, and has activities that target and 
engage other stakeholders such as community, parents, learners, BOG and PTA. 


On the other hand, schools with lower WMS scores emphasised on the benefits of scaling 
up the ‘supervision approach’ to other schools, as more frequent inspections and 
monitoring visits, and even unannounced spot-checks, create uniformity and motivate 
school staff to be more consistent in adopting best practices. 


Although the costs of the I&l inspection process are presumed to be higher than the costs 
of a regular inspection visit, the additional cost is justified by the improvements that I&l 
offers at both the school and systems-level 


Project stakeholders and school leaders agreed that the |&! pilot demonstrated value for money. In 
terms of perceived costs, PEAS and DES stakeholders recognised two components of the !&| 
inspection process: inspection visits and school improvement support. Although stakeholders 
reported that joint DES-PEAS inspection visits represented higher operational costs and greater 
logistical organisation than those of the PEAS model alone, stakeholders agreed that joint 
inspections offered other benefits. This included the opportunity to provide feedback to improve the 
new DES tool, as well as bilateral capacity building. 


According to DES stakeholders, PEAS-DES joint inspection visits could be cheaper than DES-only 
inspections since they had a geographical focus and engaged inspectors from the localities in 
which they were operating. DES also reported that the joint inspection visits were more cost- 
effective since the PEAS and DES team were able to maximise their time by observing more 
lessons and covering more areas in the school visit, and it also allowed them to prepare the 
inspections report on site, which avoided delays in implementation. The school improvement 
support component, which includes follow-up visits and calls from PEAS inspection team and 
PEAS peer head teachers, are likely to make the whole inspection process more costly than 
government inspections. Nevertheless, for DES stakeholders value for money was driven by the 
introduction of improvement plans, which closed an existing gap in DES inspection process that 
currently ends with sharing inspection recommendations to schools. 
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6 


Recommendations 


In moving forward, our evaluation findings offer recommendations for the future of the 1&I project, 
which are presented below. The recommendations are divided into implementation and partnership 
considerations. 


6.1 


Implementation 


The expectations of school leaders for the types of support offered by I&I require 
management. I&l offers a unique opportunity for school leaders to develop key 
foundational management skills as well as opportunities to build strategic management 
processes and practices. However, fundamentally school leaders face large budgetary 
constraints which may only increase as a result of the impact of COVID-19 related school 
closures. In the future, the I&! project will need to ensure that it articulates, early in the 
project, the types of support that school leaders will receive and the importance and value- 
add of professional development. The first-hand accounts and experience of school leaders 
who participated in the pilot may be useful for this, given their enthusiasm for further I&| 
scale out. 


Some school leaders would benefit from additional time and training on foundational 
management skills in order to maximise on their opportunity to learn from their 
participation in l&l. There is a differentiated impact of the 1&1 pilot for some schools, where 
some schools experienced smaller improvements to their school management practices. 
One recurring challenge for schools in these circumstances was the limited experience and 
time in role of school leaders at these schools. These were not barriers to impact per se, 
but limited the extent of impact. These school leaders would benefit from additional support 
and capacity building from I|&l to address some of these fundamental aspects of 
management skills in order that they are able to address more strategic aspects of 
management practice within the lifetime of their participation in I&l. This training or support 
could be provided by the programme or by bringing in an additional project partner to fill this 
gap. 


1&I can further its impact by building and nurturing communities of practice as part 
of professional development for school leaders. School leaders emphasised the value 
of peer mentoring (from PEAS school leaders) and individualised support and visits 
throughout the evaluation. Given the interest from school leaders to further support future 
peers, there may be an opportunity for I&! to expand its capacity to provide individualised 
support and visits through communities of practice. Where possible, this should build upon 
existing structures and professional development networks to ensure greater sustainability 
of these relationships. |&l may want to also explore other approaches to facilitate the 
sharing of experiences and knowledge within and amongst participating schools, such as 
arranging school visits or organising seminars for participants to share their successes and 
challenges. This can also reduce the dependence on PEAS and DES staff to provide 
support and mentorship and therefore offer further scalability. 
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6.2 


1&l should continue to supporting school leaders to focus on learner safety and well- 
being. Although not always a focus of school management practices, the I&l pilot was 
successful in supporting school administrators and staff to realise the importance of learner 
safety and well-being, in particular through the reduction of the use of corporal punishment. 
Comparatively, this was an area where it was straightforward to instigate change as 
respondents reported that some school staff had no knowledge about MoES’ policy that 
prohibits the use of corporal punishment in Uganda, or of alternative disciplinary methods. 
Given the additional recent experience of school closures and its ancillary effects to student 
safety and well-being, |&! should continue to promote positive discipline management and 
other training focus on learner well-being into future support to school leaders and their 
staff. 


Although not an explicit focus of the evaluation, respondents mentioned that the 
development of ICT competencies would be a beneficial addition to the professional 
development of school leaders. The evaluation did not focus on ICT elements, but when 
school leaders were asked about what further support they would benefit from, a number of 
respondents reported an interest in ICT competencies. School leaders reported that 
digitisation was a growing priority, both at the level of ICT incorporation at the curriculum 
level, as well as ICT tools to strengthen teacher management practices. 1&l may choose to 
develop and nurture partnerships to help support the development of ICT competencies as 
part of school leader professional development. This will also support I&! with the 
introduction of digitised school inspections processes. 


Partnership 


While the pilot has demonstrated that I&I offers a valuable model for school 
improvement and lessons for I&l scale out, there may need to be a further translation 
of lessons to DES’ own inspections process. Although PEAS and DES are co- 
implementers, it is important to remember that DES is the duty bearer of the inspection 
process in the country and therefore DES should also have ownership over the 
improvement process. There may need to be further efforts for DES to institutionalise 
school improvements within the enhanced school inspection approach and a fundamental 
shift is to advance from focusing on publication of inspection reports, to the provision of 
targeted support for defining school-level plans for improvement. 


Given the strong partnership and reputation of PEAS and DES, I&l has an enormous 
potential to leverage this partnership and become an advocacy platform to advance 
system-wide support for school improvement. As noted above, both PEAS and DES 
are recognised as experts on school management. With this reputation, PEAS and DES 
could leverage the I&l partnership to convene like-minded public and private organisations 
around a shared vision for school improvement in the secondary education sub-sector. This 
can help to address other prevailing challenges experienced in operations of secondary 
schools, for example infrastructure development and community participation. PEAS and 
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DES should continue to find ways to increase exposure to the programme, leveraging 
policy impact and impacts to schools outside the I&! project. In the immediate term, the 
findings and impact of the project should be shared more widely to increase the visibility 
and to facilitate embedding the programme into the government inspection process. 


e Through the experience of I&l, DES can play a key role as the champion of the 
programme, channelling evidence of impact to the MoES. As the spokesperson for I&l 
at the policy level, DES may be able to leverage the evidence from the I&I pilot and beyond 
to advocate for resources for enhanced school inspection process and school 
improvement, as well as amplifying the voice of school leadership to the government. In 
addition to integrating support for school improvement into current reforms to the routine 
school inspection process, it is important for DES to ensure that issues related to school 
improvement maintain visibility throughout MoES’ policy cycle including the next national 
education sector analysis. This presents an opportunity for the approach to be prioritised in 
future education sector plans and fosters collaboration among relevant government 
institutions around MoES’ shared vision for quality education. In particular, some 
stakeholders mentioned the importance of having the Secondary School Department at the 
MoES on board in order to build a solid foundation for sustainable changes. 
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Appendix A: Proposed I&l Theory of Change 


Theory of change for Il&l as developed by NFER at baseline 


Scale programme to have a wider, systemic impact 


Tools developed and evidence gathered to 
allow PEAS and DES to deliver wider, system- 
level impact 


Development of an effective model for 
supporting government schools to improve 
Evidence generated of practices from non-state 
sector being translated to state sector 


treatment schools improvement 


Improved school quality as measured by year 
two inspection scores 
Improved quality of school y Observable changes to areas of 
leadership and management at school practice targeted for 


Conduct 


Conduct Develop and school Final school Project evaluation ) 
school implement improvement bolpreigcte Uae 
inspections ® SIPs = low-up »> inspection 
visits 


Activities 


Adapt DES inspection tool; adapt PEAS’ inspection and school improvement 
tools and process for non-PEAS schools 
Partnership development: introductory meetings with DES 


- Intellectual property: DES inspection tool; PEAS School improvement planning and monitoring tools 
- Human resource: PEAS inspection team and school leaders; DES inspection team; PEAS UK technical team support 
- Project funding 


Inputs 
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Appendix B: Evaluation Framework 


ean ae SAME) Evaluation Sub-Question Judgement Indicators 
riteria Question 
To what extent has the quality of 
14 school management and leadership, 1144 Extent to which WMS scores have changed in treatment 
1. What impact, if “" as measured by the WMS, changed - schools 
any, did the from baseline to endline? 
intervention have 124 Changes in the school's operations as perceived by head 
on the quality of ““" teachers 
school leadership 422 Changes in the school's monitoring processes as perceived 
and management In what ways or in what areas has the a by head teachers 
in treatment 42 quality of school management and 123 Changes in the school's target setting as perceived by head 
schools? “leadership, as measured by the WMS, ay teachers 
changed from baseline to endline? 124 Changes in the school's leadership as perceived by head 
_ teachers 
125 Changes in the school's staff management as perceived by 
~ head teachers 
Through what mechanisms did the Types of mechanisms used by the intervention, as planned 
Effectiveness 2.1 intervention try to improve school 2.1.1 and in practice (e.g. HT peer support, PEAS support , HT 
leadership and management? training, Developing the SIP) 
Ways in which the intervention mechanisms have 
F ; 2.2.1 contributed to intended changes in school management and 
2. Why did the In what ways did the intervention é 
iatenventionicn ee cae Gonnbisietothé leadership (and supporting evidence of impact) 
elements of the “observed changes in school Ways in which the intervention mechanisms had unintended 
intervention management and leadership? 2.2.2 changes (positive or negative) in school management and 
demonstrate the leadership (and supporting evidence of impact) 
observed impact? 
2.3 In what ways do the intervention 23.1 Degree of alignment of the intervention mechanism and 
mechanisms align with school needs? support provided by PEAS with the needs of the school 
244 Identified barriers to improving school leadership and 
24 What other factors affected school = management during the intervention 
“" leadership and management? 242 Identified enablers to improving school leadership and 
-_ management during the intervention 
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3. What other 
areas of the 
school's 
operations, such 
as - for example - 
staff management 
or planning 
processes - if any, 
did the PEAS 
intervention affect? 


4. How much did 
the interventions 
cost to deliver and 
did they represent 
value for money? 


Efficiency 


2.5 


2.6 


3.1 


3.2 


4.1 


4.2 


In what ways have COVID-19 
disruptions created challenges for or 
supported the resilience of school 
leadership and management? 


What lessons can be learned from 
COVID-19 disruptions for the 
implementation and scale up of the 
programme? 


In what ways did the intervention have 
an effect on other areas of the school's 
operations? 


In what ways did COVID-19 
disruptions create challenges to 
quality of teaching and learning? 


How does the cost of the I&I pilot 
compare to other ways in which the 
project might have produced similar 
outcomes? 


What is the relative cost-effectiveness 
of the different intervention 
mechanisms of the intervention? 
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2.5.1 


2.5.2 


4.2.1 


4.2.2 


Identified challenges to school leadership and management 
after COVID-19 disruptions 

Ways in which intervention mechanisms have created 
resilience amongst school management in the face of 
learning disruption 


Ways in which the programme can adapt to mitigate the 
impact of any forms of learning disruption moving forward 


Observable ways in which the intervention had an effect on 
teacher attendance 

Observable ways in which the intervention had an effect on 
student attendance 

Observable ways in which the intervention had an effect on 
learner achievement 

Observable ways in which the intervention had an effect on 
learner behaviour 

Observable ways in which the intervention had an effect on 
learner safety 

Identified challenges (and changes in challenges) to 
improving the quality of teaching and learning after COVID- 
19 disruptions 


Ways in which COVID-19 learning disruptions have 
changed schools’ improvement objectives moving forward 
Cost of delivering each intervention mechanism in treatment 
schools (in terms of estimated staff costs and expenses) 


Cost of partnership between DES and PEAS (versus DES 
or PEAS alone) 


Perceived effectiveness of each intervention mechanism to 
improve school leadership and management 


Relative cost of different intervention mechanisms (JI 4.1.1.) 
versus their level of effectiveness to bring improvements to 
school management and leadership (Jl 4.2.1) 
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4.3 


5. What lessons 

can be learned 5.2 
from the pilot 

about how the 
intervention can be 
scaled out? 5.3 


Sustainability 


5.4 


How valuable to school leaders and 
PEAS staff were the changes brought 
about by the intervention, relative to 
the time and money invested? 
Relevance: How has the intervention 
targeted and addressed the barriers to 
school improvement in Uganda? 
Feasibility: To what extent was it 
feasible for PEAS and DES to 
implement the intervention? 


Adaptability: How was the programme 
adapted without negatively affecting 
outcomes? 


Scalability: To what extent is there 
political will and institutional/ financial 
capacity to scale up the programme? 


4.3.1 


5.1.1. 


5.2.1. 


5.3.1 


5.3.2 


5.4.1 


Perceived value of the intervention in improving 1) school 
management and leadership and 2) quality of teaching and 
learning, relative to the intervention costs 


Ways in which the intervention has addressed the 
established and perceived barriers to school improvement 
in Uganda 


School-level enablers and barriers in implementing the 
intervention in treatment schools 


Ways in which PEAS has adapted the intervention 
mechanisms in order to be implemented in treatment 
schools 

Ways in which lessons learned are incorporated into the 
programme 


Systems-level enablers and barriers in implementing the 
intervention treatment schools 
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Appendix C: 


List of Respondents 


Data collection 
method 


Role 


School 


Basic school 
survey and 
WMS 


Head Teacher 


Priscilla Comprehensive Girls' S.S 


Deputy Head teacher 


Toroma Secondary School 


Head Teacher 


Dr. Aporu Okol Memorial 
Secondary school 


Head Teacher 


Katakwi High School 


Head Teacher 


Magoro Comprehensive 
Secondary School 


Head Teacher 


Ococia Girls S.S 


Head Teacher 


Kuju Seed Secondary School 


Deputy Head teacher 


St.Francis S.S Acumet 


Head Teacher 


Mukongoro High School 


Director of Studies 


Kanyum Comprehensive SS 


Data collection 
method 


Key stakeholder 
Interviews 


Role Institution / School 
Head of Quality Assurance PEAS 

Education and Partnerships Lead PEAS 

Senior Inspection of Schools PEAS 

Peer school leader mentor PEAS 
Commissioner Secondary Education DES 

Standards 

District Inspector of Schools, Kumi DES 

District Local Government 

Assistant Commissioner Education DES 


Standards 


Case study 1 
Interviews 


Director of Studies 


Deputy Head teacher 


School treasurer (member of Board of 
Governors) 


Chairman of Finance Committee 
(member of Board of Governors) 


Assistant Director of Studies 


Kanyum Comprehensive 
Secondary School 


Case study 2 
Interviews 


Head Teacher 


Head of CRE and History Department 


Head of ICT 


Member of the Board of Governors 


Teacher of physics and mathematics 


St. Francis Acumet Secondary 
School 
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Deputy Head Teacher 


Director of Studies 


Interviews Secondary School 
Head of Science Department 


Deputy Vice Chairperson Board of 
Governors 


Head Teacher 


Deputy Head teacher 


Case study 4 


intaniowe Senior teacher Ococia Girls Secondary School 


Vice Chairperson Board of Governors 


Head of Arts department 
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Appendix D: Research Instruments Used at Endline 


Appendix D1: Basic school survey and WMS tool 
Evaluation of PEAS I&I Pilot: Head teacher survey 


(Vere | Tfat=9) 


Note to enumerator: Please read out loud the preamble below to the respondent and record 
the answer to the consent question below. Then complete the survey details section below, 
asking the questions as required. Please then detach this cover sheet from the remainder of the 
survey. 


This survey is part of a research project conducted by the National Foundation for Educational 
Research (NFER), with support from Ichuli. NFER is a UK-based charity whose mission is to 
generate evidence and insight that can support positive change across education systems. 


You were selected to participate in this survey as you are a leader in one of the schools where 
the NGO PEAS, in partnership with DES, is implementing its Schools Inspect & Improve 
project, and your school participated in the baseline survey in 2019. We would like to gather 
some information about school management and leadership and about your participation in the 
project. 


Please note that there are no right or wrong answers, so please answer as honestly as 
possible. All of the answers you give will be confidential and will not be shared with anyone 
other than members of the research team from Ichuli and NFER. You will remain anonymous 
in any analysis that is presented. 


If you wish to stop the survey at any time, please let me know. If at any time in the future you 
want to withdraw your answers, you can contact us using the details on the card provided, and 
we will remove your data. 


In the first part of the survey | will ask you about different areas of the school leadership and 
management and how they work on your school. This will take approximately 60 minutes. 
Following this, | will ask you a set of questions with answer options for you to choose. This last 
section will cover more specific aspects of your participation in the project and will take 
approximately 15 minutes. Are you happy for us to proceed? 


Note to enumerator: Please record the consent provided in the boxes below. 


L] Yes [_INo 


Suspend survey if response is ‘no’. 


Survey details 


1. School name and code 


Participant name 
Participant gender L_] Male [_] Female | (] Not sure 


Participant position 


a); )eo)p 


Participant contact information (email and 
alternative phone number if available) 


Date of interview 


Start time 
End time 
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Section 1 —- Management survey: 60 min 


In this first section of the survey | will ask you about management practices in your school. To answer these questions, please focus on the school’s ways 
on working in the period before the COVID-19 school closures in March 2020, that is the second and third term of the 2019 school year and the start of 
the first term of the 2020 school year. | will ask you about general areas related to operations, planning, best practices, monitoring, accountability and 
people management, among others. You may find these questions familiar since we delivered this survey in March 2019, as part of the baseline data 
collection for the evaluation. 


Note to enumerator: The following section contains 12 areas for which you must rank the respondent’s school based on their responses and the scoring 
matrix provided, using the scoring guide below. Start by telling the respondent what area you will be asking questions about (the bolded probe on the left 
side) and then use the open-ended probes on the right. Do not read out the scoring guide or share with the respondents that they are being scored. Try 
to spend approximately five minutes on each of the 12 areas only. Use a score of 99 if the respondent is not aware of the answer or does not respond. 


Guide and scoring matrix 


OPERATIONS 
1) Standardisation of instructional How do you ensure that all teachers cover all the curriculum topics? 
processes Do teachers make lesson plans or schemes of work? 


Are these schemes or plans fixed at the start of term or do they change throughout the year? 
How do you keep track of what teachers are doing in the classrooms? 


2252 


First, | would like to know how 
standardised teaching processes 
are in your school. 


Score 1: No clear planning Score 3: School has defined process | Score 5: School has a defined 


processes or protocols exist; little 
verification or follow up is done to 


for developing schemes of work and 
lesson plans; they are prepared by 


Score: ensure consistency across all teachers and checked at continually adapt their lesson plans, 
classrooms beginning of term; they have some based on student performance; 
10 20 30 40 5n 99 flexibility to meet student needs; there is a regular dialogue with 


monitoring is only adequate (i.e. a 
few times throughout term) 


process for developing schemes of 
work; all teachers are encouraged to 


teacher and senior management 
about lesson content /curriculum 
coverage. 
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2) Personalization of instruction and 
learning 


Next, | would like to know about 
how flexible teaching methods 
are. 


a) How much does the school attempt to identify individual student needs? How are these needs 
accommodated for within the classroom? 
b) How do you as a school leader ensure that teachers are effective in personalising instruction in each 


classroom across the school? 


c) What about students, how does the school ensure they are engaged in their own learning? How are parents 


incorporated in this process? 


Score 1: Teachers lead learning with 
very low involvement of students; 
there is little or no identification of 
diverse student needs. Booster 
classes are not targeted 


Score 3: There are remedial 
lessons, There is some evidence of 
in-class strategies to make sure all 
students are learning in classroom 
and actively participating (Such as 
through group work, continuous 
assessment) 


Score 5: Emphasis is placed on 
personalization of instruction based 
on student needs; school 
encourages student involvement and 
participation in classrooms; school 
provides information to and connects 
students and parents with sufficient 
resources to support student 
learning 


3) Data-driven planning 
! would like to understand if and 


how your school uses data to 
make decisions. 


Score: 


1o 20 30 40 50 99 


a) What type of information on each individual student’s ability is available to teachers at the beginning of the 


academic year? 


b) How do you track the progress of each student throughout the year? 


c) Howis data used by teachers? 


Score 1: Little or no effort is made to 
provide new teachers with 
information about students as they 
move through the school; no culture 
of reviewing student data throughout 
the year. 


Score 3: Schools may understand 
the importance of tracking student 
performance as they move through 
school, but they do not have 
consistent processes in place. Some 
data available throughout the year 
but not easy to interpret or 
understand, and will sometimes 
inform teaching practice (i.e. re- 
teaching a topic) 


Score 5: Students progress is 
managed in an integrated and 
proactive manner, supported by 
formative assessments tightly linked 
to expectations; data is widely 
available and easy to use 
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4) Adopting and leading on 
education best practices 


! would like to understand how 
you and your teachers work on 
improving the quality of teaching 
and learning in your school. 


Score: 


1o 20 30 40 55 99 


HELO FH 


oO 


How do teachers learn about new education best practices? 
How do you encourage the teachers to incorporate new teaching practices into the classroom? 
Do you or other members of staff observe any classes? How often? 
How do you support teachers in improving their practice after observations? 
How are teachers trained on best practices around student safety? 


Score 1: Teachers do not go on 
courses and there is no convincing 
explanation of how teachers are 
encouraged to improve their 
teaching or ensure the safety of 
students. Classes are not observed 
by senior management 


Score 3: Teachers go on training 
courses, and are encouraged to 
adopt new techniques. School 
leaders ensure teachers are aware 
of student safety and discourage use 
of corporal punishment. There is a 
proper system for them to share the 
learnings of the training with their 
colleagues and some monitoring 
afterwards. School leaders observe 
some classes, and some feedback is 
provided 


Score 5: Teachers go on training, 
share with colleagues, are followed 
up on, and get additional school 
based training from senior staff. 
Strong culture of learning and 
sharing amongst the staff. School 
leaders set high expectations for 
teaching practice and student safety, 
and regularly observe classes and 
provide feedback for support 


MONITORING 


5) Continuous improvement 
! will now ask you about the 


problem-solving techniques used 
in your school. 


Score: 


1o 20 30 40 50 99 


a) When problems (e.g. within school/ teaching tactics/ etc.) do occur, how do they typically get exposed and 


fixed? 


b) Can you talk me through the process for a recent problem that you faced? 
c) Who within the school gets involved in changing or improving process? How do the different staff groups 


get involved in this? 


d) Does the staff ever suggest process improvements? 


Score 1: Exposing and solving 
problems (for the school, individual 
students, teachers, and staff) is 
unstructured; no process 
improvements are made when 
problems occur, or there is only one 
staff group involved in determining 
the solution 


Score 3: Exposing and solving 
problems (for the school, individual 
students, teachers, and staff) is 
approached in an ad-hoc way; 
resolution of the problems involves 
most of the appropriate staff groups 


Score 5: Exposing and solving 
problems (for the school, individual 
students, teachers, and staff) ina 
structured way is integral to 
individual's responsibilities, and 
resolution involves all appropriate 
individuals and staff groups; 
resolution of problems is performed 
as part of regular management 
processes 
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6) Performance tracking, review and 
dialogue 


Tell me about how school 
performance is tracked and 
reviewed in your school. 


Score: 


1o 20 30 40 55 99 


a) What kind of main indicators do you use to track school performance? What sources of information are 


used to inform this tracking? 


b) How frequently are these measured? Who gets to see this performance data? 
c) If | were to walk through your school, how could | tell how it was doing against these main indicators? 


Score 1: Measures tracked do not 
indicate directly if overall objectives 
are being met, tracking is an ad-hoc 
process (certain processes are not 
tracked at all) 


Score 3: Most performance 
indicators are tracked formally; 
tracking is overseen by the school 
leadership only 


Score 5: Performance is 
continuously tracked and 
communicated, both formally and 
informally, to all staff using a range 
of visual management tools 


TARGET SETTING 


7) Balance of targets/metrics 


! want to ask you about target 
setting in your school. 


Score: 


1o 20 30 40 50 99 


Do you have any targets? 


) 

) Are your targets linked to student outcomes? 

) Can you tell me about any specific goals for departments or teachers? 
) Can you tell me about any specific goals for students? S1-S3, and S4? 
) How do you decide what targets to set? 


Score 1: Performance targets are 
very loosely defined or not defined at 
all; targets set only include those 
required by the DES / other authority 


Score 3: Performance metrics and 
targets are defined for the school 
and teachers, based on student 
results. S4 students should also 
have individual targets; targets set 
include some consideration beyond 
what is required 


Score 5: Performance measures are 
defined for all, and they include 
measures of student outcomes and 
other important factors linked to 
outcomes (i.e. attendance). When 
they are combined, specific short- 
term targets are designed to meet 
long term goals. Targets set go 
beyond what is required and 
considers specific school needs or 
goals 


8) Target stretch 


! want to understand about your 
progress related to your targets. 


Score: 


1o 20 30 40 50 99 


) Do you usually meet your targets? 
b) How do you decide how difficult to make your targets? 
) Do you feel that all the department/areas have targets that are just as hard? Or would some 


areas/departments get easier targets? 


Score 1: Goals are either too easy or 
impossible to achieve; at least in 
part because they are set with little 
teacher involvement or no use of 
data. OR: No goals are set 


Score 3: Some carefully considered 
and clear targets in place, taking into 
account some evidence; targets 
dictated by head teacher, with little 
buy-in from teachers and limited 
external benchmarking. 


Score 5: Targets are genuinely 
demanding whilst still realistic for all 
parts of the organisation; goals are 
set in consultation with senior staff, 
and consider external benchmarks 
where appropriate. 
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9) Budgeting 


! would like to understand the way 
budgeting is done in your school. 


Score: 


1o 20 30 40 55 99 


a) Do you prepare a budget for the school? 
b) How do you plan the spending of your budget? 
c) How do you make sure you don’t overspend or underspend each year? 


Score 1: No clear process for 
preparing or monitoring budgets 


Score 3: Clear process for preparing 
budgets, some monitoring 
throughout the year, some possibility 
to reforecast to cover 
over/underspends 


Score 5: Clear process for preparing 
budgets; different scenarios planned 
for (increase/decrease in income); 
budget regularly reviewed by senior 
leadership; process for 
reforecasting; effectively managed to 
avoid overspend 


LEADERSHIP 


10) Leadership vision 


! will now ask you about the 
school’s vision. 


Score: 


1o 20 30 40 50 99 


overall school community? 


) What is the school’s vision for the next five years? Do teachers/ staff know and understand the vision? 
b) Who does your school consider to be your key stakeholders? How is this vision communicated to the 


c) Whois involved in setting this vision/ strategy? When there is disagreement, how does the school leader 


build alignment 


Score 1: School either has no clear 
vision, or one defined without 
substantial stakeholder collaboration 
and which focuses primarily on 
meeting state/ national mandates; 
school leader does not or cannot 
articulate a clear focus on building 
an environment conducive to 
learning 


Score 3: School has defined a vision 
that focuses on improvement in 
student outcomes, but largely 
focused on meeting state/ national 
mandates, and usually defined with 
limited stakeholder collaboration; 
school leaders may focus on the 
quality of the overall school 
environment, but often in response 
to specific issues 


Score 5: School leaders define and 
broadly communicate a shared 
vision and purpose for the school 
that focuses on improving student 
learning (often beyond those 
required by law); vision and purpose 
is built upon a keen understanding of 
student and community needs, and 
defined collaboratively with a wide 
range of stakeholders; school leader 
proactively builds environment 
conducive to learning 
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11) Clearly defined accountability for 
school leaders 


! now want to understand more 


about accountability in your 
school. 


Score: 


1o 20 30 40 55 99 


Who is accountable for delivering on school targets? 


cost targets as well as quality targets? 


) 
b) How are individual school leaders held responsible for the delivery of targets? Does this apply to equity and 


c) What authority do you have to impact factors that would allow them to meet those targets (e.g. budgetary 


authority, hiring & firing)? Is this sufficient? 


Score 3: School leaders are held 
accountable for absolute number of 
student reaching targets set by 
government and school internally, 
with school-level and individual 
consequences for good and poor 
performance; school leaders are 
provided some autonomy to impact 
the areas of accountability 


Score 1: School leaders are only 
held accountable for minimal targets 
(e.g. those set by government), 
without school- level or individual 
consequences for good and poor 
performance; school leaders have 
little or no autonomy to impact the 
areas of accountability 


Score 5: School leaders are held 
accountable for quality, equity and 
cost- effectiveness of student 
outcomes within the school, with 
school-level and individual 
consequences for good and poor 
performance; school leaders are 
provided sufficient autonomy to 
impact the areas of accountability 


PEOPLE MANAGEMENT 


12) People Management 


! now want to understand about 
how you manage your teachers. 


Score: 


1o 20 30 40 50 99 


Who makes decisions to hire teachers? 

Who determines the recruitment criteria? 

How do you know who are your best teachers? 

What types of rewards are offered to teachers? 

How do you keep the best teachers motivated? 

How do you know who are the teachers who are not doing so well? 


SFeEAacew 


Score 3: Schools refer to centralised 
authorities to recruit and deploy 
teachers but can send requests for 
resourcing. Performance 
management is dictated by 
centralised authorities 


Score 1: School has no formal 
control over recruitment and 
rewards. There is limited or no 
structured way of monitoring the 
performance of teachers 


Score 5: School proactively works 
with centralised authorities to recruit 
and retain the best teachers. 
Teachers undergo formal and 
informal reviews regularly. 
Motivational rewards are available in 
the school. 
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Section 2 — Pilot implementation and effectiveness survey: 15 min 


In this second and last section of the survey | will ask you more specifically about your 
participation in the project and whether you think it was effective in achieving improvements in 
school management and quality of teaching and learning. Unless stated otherwise, the 
questions are focused on the period between the PEAS inspection in (Mar/June) 2019 up 
until school closures in March 2020, and a few questions will relate to the challenges of 
learning disruption due to the COVID-19. | will ask you a set of questions and give you answer 
options to choose from, or in some cases | will ask you to give a score. 


Note to enumerator: Please read the following questions and answer options aloud to 
participants. Please tick ONE box that best describes the participant’s answer, unless the 
question specifies to tick more than one box. If the respondent does not know the answer or is 
non-respondent, please note ‘99’ into the answer box. If further space is required to record 
additional details, please use the back of the page. Alternatively, you can record the answers 
directly on the Excel Spreadsheet provided. 


Question Record answer here 
1. For how many years/months Years 
have you been in this post as a Months 


school leader in this school? 


2. How many students were 
enrolled at this school at the 
beginning of the 2019 school 
year? 

3. How many students are 
currently enrolled at this 
school? (Feb 2021) 

4. How many of the students that 
are currently enrolled are 
girls? (Feb 2021) 

5. What type and approximately Read these options out loud 


how much support did you 5.1. Training from PEAS 
receive as part of the project? | yes CINo 


If yes, how many sessions? 


5.2. Peer support from other head teachers (other l&lI 
schools and PEAS schools) 


[ ]Yes [ ]No 


If yes, how many calls? 
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How many visits? 


5.3. Support from PEAS staff 
LlYes L_JNo 


If yes, how many calls? 


How many visits? 


[_] Other, please specify the type of support 


6. Thinking about the different Read these options out loud and tick one box in each case. 


Was I which: MENG DrOviGed 6.1. Development of a School Improvement Plan SIP 


support to you, to what extent (J Not at all effective 
do you think each of the 


following strategies was 
effective in improving school L_] Moderately effective 
leadership and management? 


[_] Somewhat effective 


[_] Completely effective 


| am going to read out four different |6.2. School leader’s training 
support strategies and | want you to | L] Not at all effective 

rate them according to the 
response options | will give you. 


L_] Somewhat effective 
L_] Moderately effective 
[_] Completely effective 


6.3. Head teacher peer support 
L] Not at all effective 


L_] Somewhat effective 
L_] Moderately effective 
[_] Completely effective 


6.4. PEAS staff support 
L] Not at all effective 


L_] Somewhat effective 
L] Moderately effective 
[_] Completely effective 


_] Other, please specify the strategy 
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7. In what areas of the School 
Improvement Plan did you 
notice positive changes? | am 
going to read seven areas and | 
would like you to tell me how 
much progress you made. 


A score of 1 represents ‘no 
observable changes’ and 5 is 
‘improvement objectives were met’. 


Read the options out loud and write the score from 1 to 5. 


7.1... Teacher attendance 

7.2. Learner attendance 

7.3. Learner achievement 

7.4. Teaching quality 

7.5. Learner behaviour 

7.6. Learner safety a 
7.7. School leadership and governance 


8. To what extent do you think the 
improvement objectives and the 
strategies set in the SIP were 
aligned to the needs of your 
school? 


Read these options out loud: 
L] Not at all aligned 

L_] Somewhat aligned 

[_] Moderately aligned 

[_] Completely aligned 


9. To what extent did you and your 
staff have the capacity to 
implement the activities set out 
in the SIP in your school? 


Read these options out loud: 
L_] Not at all 

L_] Somewhat 

L_] Moderately 


you received other assistance 
to support school leadership? 


For example, this might include 
other training or accountability 
programmes from NGOs or the 
government. 


[_] Completely 
10. = Apart from the national L] Yes 
education system support, have CI No 


If yes, please specify what kind of support was received. 


11. Thinking about the period 
since schools reopened after 
the COVID-19 closures (from 
October 2020 until this day), 
what are the three key 
challenges you are facing in 
your school that have an impact 
on the quality of teaching and 


Read these options out loud. Tick three. If answer option 


desired is not available, record this under other and specify 


the challenge. 
|] Teachers are absent 


[_] Professional development support for teachers is 
inadequate 
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learning? 


12. To what extent did the 
changes in quality of 
school leadership and 
management justify the 
time and resources that 
you have invested in 
participating in the 
project? 


[_] Teachers are not motivated 

[_] Students are absent 

_] Students are dropping out of school 

[_] Students are unmotivated or distressed 

_] Learning loss due to school closures 

L_] Local attitudes affecting girls’ access to education 
_] Inadequate school infrastructure 

_] Inadequate teaching materials 

L_] Parental support is missing 


L_] School budget is inadequate 


L_] Other 


Read these options out loud: 
L_] Not at all 

L_] Somewhat 

[_] Moderately 

[_] Completely 


13. To what extent did the 
changes in quality of 
teaching and learning 
justify the time and 
resources that you have 
invested in participating 
in the project? 


Read these options out loud: 
L_] Not at all 

L_] Somewhat 

[_] Moderately 

[_] Completely 


14. What challenges to 
school leadership and 
management have been 
created or accentuated 
by COVID-19 
disruptions? 


Read these options out loud. Tick up to three. If answer 
option desired is not available, record this under other 
and specify the challenge. 


[_] Focus on short-term planning 

_] Tighter reporting to authorities 

L_] Little autonomy over decision making 

[_] Lack of collaboration with local entities (e.g. PTA’s) 


[_] Destandardisation of teaching practices 
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[_] Changes in teacher’s job satisfaction 


[_] Changes in teacher's attitudes towards school 
leadership 


[_] Deprioritised areas of curriculum 


L_] Insufficient capacity to implement re-opening safety 
guidelines 


[_] No challenges created 


of COVID-19 disruptions 
in the school? 


[_] Other 
15. To what extent has Read these options out loud: 
participation in the L_] Not at all 
PEAS-DES project given [_] Somewhat 
you leadership and 
management tools to LJ Moderately 
help mitigate the impact | [_] Completely 


If answered moderately or completely, could you 
describe to me in one sentence how the pilot helped 
you to handle COVID-19 challenges? 
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Appendix D2: Interview Guides 


Head Teacher interview guide 


Headteacher 
Interview Guide_PEAS 


School Staff interview guide 


School staff KI 
Guide_PEAS Evaluatior 


Project Stakeholders’ PEAS interview guide 


Project Stakeholders’ 
PEAS Interview Guide . 


Project Stakeholders’ DES interview guide 


Project Stakeholders’ 
DES Interview Guide_| 
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Appendix E: WMS Scoring Rubric 


Topic 


Questions 


Scoring Rubric 


3 


Operations 


1. Standardisation 
of instructional 
processes 


Tests how well 
materials and 
practices are 
standardised and 
aligned in order to 
be capable of 
moving students 
through learning 
pathways over 
time 


a) How do you ensure that 
all teachers cover all the 
curriculum topics? 

b) Do teachers make 
lesson plans or schemes of 
work? 

c) Are these schemes or 
plans fixed at the start of 
term or do they change 
throughout the year? 

d) How do you keep track 
of what teachers are doing 
in the classrooms? 


« No clear planning 
processes or protocols 
exist 

* Little verification or 
follow up is done to 
ensure consistency 
across classrooms 


* Schemes of work 
prepared by all teachers 
and checked at beginning 
of term 

+ Likely to be some 
lessons plans in place 

* There is no flexibility to 
meet student needs 

¢ There is little monitoring 
throughout the term 


* School has defined 
process for developing 
schemes of work and 
lesson plans 

« Lesson plans are 
prepared by all teachers 
and checked at beginning 
of term 

* They have some 
flexibility to meet student 
needs 

* Monitoring is only 
adequate (i.e. a few 
times throughout term) 


* School has defined 
process for developing 
schemes of work and 
lesson plans 

* Lesson plans are 
prepared by all teachers 
and checked at beginning 
of term 

* Teachers are 
encouraged to adapt 
some lesson plans 
throughout the term 

¢ There frequent 
monitoring through 
different means (i.e. 
lesson observations, 
checking student books) 


* School has a defined 
process for developing 
schemes of work and 
lesson plans 

« Lesson plans are 
prepared by all teachers 
and checked at beginning 
of term 

¢ All teachers are 
encouraged to continually 
adapt their lesson plans, 
based on student 
performance 

¢ There is a regular 
dialogue with teacher and 
senior management 
about lesson 
content/curriculum 
coverage 


2. Personalisation 
of instruction and 
learning 


Tests for flexibility 
in teaching 
methods and 
student 
involvement 
ensuring all 
individuals can 
master the 
learning 
objectives 


a) How much does the 
school attempt to identify 
individual student needs? 
How are these needs 
accommodated for within 
the classroom? 

b) How do you as a school 
leader ensure that teachers 
are effective in 
personalising instruction in 
each classroom across the 
school? 

c) What about students, 
how does the school 
ensure they are engaged in 
their own learning? How 
are parents incorporated in 
this process 


* Teachers lead learning 
with very low involvement 
of students 

* There is little or no 
identification of diverse 
student needs 

¢ Booster classes are not 
targeted 


* There is limited active 
participation of students 
in lessons 

¢ There are some 'add- 
on’ strategies put in place 
to support struggling 
students, such as 
targeted booster classes 
or remedial lessons 
(rather than embedded 
within lessons) 


* There are remedial 
lessons 

* There is some evidence 
of in-class strategies to 
make sure all students 
are learning in classroom 
and actively participating 
(such as through group 
work, continuous 
assessment) 


* There are a range of 
techniques used to 
differentiate instruction 
and promote active 
participation in learning 

¢ There is work to ensure 
that lessons are 
appropriate for a range of 
student groups within in 
the class 


« Emphasis is placed on 
personalization of 
instruction based on 
student needs 

* School encourages 
student involvement and 
participation in 
classrooms 

* School provides 
information to and 
connects students and 
parents with sufficient 
resources to support 
student learning 
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Topic 


Questions 


Scoring Rubric 


3 


Operations 


3. Data-driven 
planning 


Tests if the school 
uses assessment 
to verify learning 
outcomes at 
critical stages, 
making data 
easily available 
and adapting 
student strategies 


a) What type of information 
on each individual 
student’s ability is available 
to teachers at the 
beginning of the academic 
year? 

b) How do you track the 
progress of each student 
throughout the year? 

c) How is data used by 
teachers? 


¢ Little or no effort is 
made to provide new 
teachers with information 
about students as they 
move through the school 
¢ There is no culture of 
reviewing student data 
throughout the year 


* Schools track some 
performance data about 
students 

¢ Data is tracked, but it is 
not frequent (perhaps 
only at the end of the 
year or term) 

* Data is often not easy 
to use and it is often high 
level (i.e. pass or fail) 


* Schools may 
understand the 
importance of tracking 
student performance as 
they move through 
school, but they do not 
have consistent 
processes in place 

* Some data available 
throughout the year but 
not easy to interpret or 
understand 


* Data is used regularly 
to guide planning and 
teaching (not just the 
typical BoT, MoT and 
EoT exams) 

* Data is used to 
understand areas of 
strength and weakness, 
and teaching is adapted 
on the basis of this 
information 


* Students progress is 
managed in an integrated 
and proactive manner, 
supported by formative 
assessments tightly 
linked to expectations 

* Data is widely available 
and easy to use 


teaching best 
practices and the 
sharing of these 
resources into the 
classroom 


classroom?c) Do you or 
other members of staff 
observe any classes? How 
often?d) How do you 
support teachers in 
improving their practice 
after observations?e) How 
are teachers trained on 
best practices around 
student safety? 


the safety of studentss 
Classes are not observed 
by senior management: 
No support is given to 
teachers to improve their 
practice 


approaches, but no clear 
system for sharing or 
monitoring improved 
practices* School leaders 
do not explicitly address 
student safety 


training with their 
colleagues and some 
monitoring afterwards: 
School leaders observe 
some classes, and some 
feedback is provideds 
School leaders ensure 
teachers are aware of 
student safety and 
discourage use of 
corporal punishment 


senior management to 
ensure training is being 
used There is some 
culture of learning and 
sharing best practices 
amongst the staffs School 
leaders set expectations 
for student safety and 
discourage use of 
corporal punishment 


accordingly * Data will sometimes 

inform teaching practice 

(i.e. re-teaching a topic) 
4. Adopting a) How do teachers learn * Teachers do not go on * Teachers go on training | * Teachers go on training | * Teachers use a range * Teachers go on 
education best about new education best courses: There is no courses (i.e. SESMAT, courses, and are of techniques to find out training, share with 
practices Tests practices?b) How do you convincing explanation of | government refresher encouraged to adopt new | about best practice, and colleagues, are followed 
how well the encourage the teachers to how teachers are courses, etc.)* The techniquess There is a share with colleagues in up on, and get additional 
school incorporate new teaching encouraged to improve school actively tries to proper system for them to | the schools There is school-based training 
incorporates practices into the their teaching or ensure take forward new share the learnings of the | regular monitoring by the | from senior staffs There 


is a strong culture of 
learning and sharing 
amongst the staff« 
Management regularly 
observe classes and 
provide feedback for 
supports School leaders 
set high expectations for 
teaching practice and 
student safety 
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Topic 


Questions 


Scoring Rubric 


3 


Monitoring 


5. Continuous 
improvement 


Tests attitudes 
towards 
continuous 
improvement 


a) When problems (e.g. 
within school/ teaching 
tactics/ etc.) do occur, how 
do they typically get 
exposed and fixed? 

b) Can you talk me through 
the process for a recent 
problem that you faced? 

c) Who within the school 
gets involved in changing 
or improving process? How 
do the different staff groups 
get involved in this? 

d) Does the staff ever 
suggest process 
improvements? 


« Exposing and solving 
problems (for the school, 
individual students, 
teachers, and staff) is 
unstructured 

* No process 
improvements are made 
when problems occur, or 
there is only one staff 
group involved in 
determining the solution 


¢ There is not a clear and 
consistent process for 
identifying and solving 
problems 

* School leaders 
sometimes involve a 
range of people and put 
in place a considered 
solution, but other 
problems are not treated 
in a structured way 


* Exposing and solving 
problems (for the school, 
individual students, 
teachers, and staff) is 
approached in an ad-hoc 
way 

* Resolution of the 
problems involves most 
of the appropriate staff 
groups 


* There is a structured 
process for solving a 
problem 

¢ Appropriate people are 
involved and school 
leaders are proactive to 
find and expose 
problems 

* Mechanisms are in 
place to learn the lessons 
from problems that have 
arisen 


« Exposing and solving 
problems (for the school, 
individual students, 
teachers, and staff) ina 
structured way is integral 
to individual's 
responsibilities 

¢ Resolution involves all 
appropriate individuals 
and staff groups 

« Resolution of problems 
is performed as part of 
regular management 
processes 


6. Performance 
tracking, review 
and dialogue 


Tests whether 
school 
performance is 
measured, 
reviewed and 
discussed and 
followed up with 
the right 
frequency, and to 
a high quality 


a) What kind of main 
indicators do you use to 
track school performance? 
What sources of 
information are used to 
inform this tracking? 

b) How frequently are 
these measured? Who 
gets to see this 
performance data? 

c) If | were to walk through 
your school, how could | 
tell how it was doing 
against these main 
indicators? 


¢ Measures tracked do 
not indicate directly if 
overall objectives are 
being met 

* Tracking is an ad-hoc 
process (certain 
processes are not 
tracked at all) 


*« Some useful 
performance indicators 
are tracked, including on 
students’ academic 
performance 

¢ Indicators are based on 
accurate but limited data 
¢ Reviews are confined to 
the senior management 
and can be superficial, 
without stimulating any 
action 


* Most performance 
indicators are tracked 
formally 

¢ Tracking is overseen by 
the school leadership 
only 


¢ Performance indicators 
are regularly tracked, and 
reviews involve a range 
of relevant staff 

* They are clearly 
documented and made 
visible to key 
stakeholders for 
example, the community 
* Attempts are made to 
use review meetings to 
solve problems 


¢ Performance is 
continuously tracked and 
communicated, both 
formally and informally, to 
all staff using a range of 
visual management tools 
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Scoring Rubric 


Topic Questions 1 2 3 4 5 
Target setting 
7. Balance of a) Do you have any * Performance targets * Performance targets for | * Performance metrics * Performance metrics « Performance measures 
targets/metrics targets?b) Are your targets | are very loosely defined the school exist but are and targets are defined are defined for the school | are defined for all, and 


Tests whether the 
system tracks 
meaningful 
targets tied to 
student outcomes 
and whether this 
is approach is 
rational and 
appropriate 


linked to student 
outcomes?c) Can you tell 
me about any specific 
goals for departments or 
teachers?d) Can you tell 
me about any specific 
goals for students? S1-S3, 
and S4? 


or not defined at alle 
Targets set only include 
those required by the 
DES / other authority 


high level; for example 
they are usually confined 
to number of Division 1 
grades in the school, or 
not linked directly to 
outcomes (such as 
attendance or 
enrolment): Targets set 
only include what is 
required 


for the school and 
teachers, based on 
student results* S4 
students should also 
have individual targets 
Targets set include some 
consideration beyond 
what is required 


and teachers based on 
student results* Targets 
are in place for all 
students, including those 
in S1-S3+ Targets are 
based on good data 
about on-going student 
performance, such as 
robust end of term tests- 
Targets set include go 
beyond what is required 


they include measures of 
student outcomes and 
other important factors 
linked to outcomes (i.e. 
attendance)* When they 
are combined, specific 
short-term targets are 
designed to meet long 
term goalss Targets set 
go beyond what is 
required and considers 
specific school needs or 
goals 


8. Target Stretch 


Tests whether 
targets are 
appropriately 
difficult to achieve 


a) Do you usually meet 
your targets? 

b) How do you decide how 
difficult to make your 
targets? 

c) Do you feel that all the 
department/areas have 
targets that are just as 
hard? Or would some 
areas/departments get 
easier targets? 


* Goals are either too 
easy or impossible to 
achieve, at least in part 
because they are set with 
little teacher involvement 
or no use of data 

* OR: No goals are set 


* Some targets are put in 
place based on 
consideration of a limited 
range of relevant data, 
such as looking at ability 
of current cohort and 
previous year’s results 

* No benchmarking 


* Some carefully 
considered and clear 
targets in place, taking 
into account some 
evidence 

* Targets dictated by 
head teacher, with little 
buy-in from teachers and 
limited external 
benchmarking 


* Targets are in place, 
based on a range of 
evidence, including some 
external benchmarks 

* Targets are adapted for 
different parts of the 
school (for example, in 
particular subjects or for 
particular student 
cohorts), rather than a 
‘one-size fits all’ 
approach 


* Targets are genuinely 
demanding whilst still 
realistic for all parts of the 
organisation 

* Goals are set in 
consultation with senior 
staff, and consider 
external benchmarks 
where appropriate 


9. Budgeting 


Tests whether the 
school has 
processes for 
planning, 
monitoring and 
adjusting their 
budgets 


a) Do you prepare a 
budget for the school? 

b) How do you plan the 
spending of your budget? 
c) How do you make sure 
you don’t overspend or 
underspend each year? 


¢ There is no clear 
process for preparing or 
monitoring budgets 


* There is some process 
for preparing budgets 
with some link to school 
needs 

* Monitoring is limited 
throughout the year 


* Clear process for 
preparing budgets 
* Some monitoring 
throughout the year 
* Some possibility to 
reforecast to cover 
over/underspends 


¢ Clear process for 
preparing budgets 

* Budgets are monitored 
throughout the year 

¢ Different scenarios are 
planned for 


* Clear process for 
preparing budgets 

¢ Budget is regularly 
reviewed by senior 
leadership 

¢ Different scenarios are 
planned for 
(increase/decrease in 
income) 

* There is a process for 
reforecasting 

* Budgets are effectively 
managed to avoid 
overspend 
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Topic 


Questions 


Scoring Rubric 


3 


Leadership 


10. Leadership 
visionTests 
whether school 
leaders have an 
understanding of 
the broader set of 
challenges that 
the school, 
system and key 
actors face and 
the right mind-set 
to address them 


a) What is the school’s 
vision for the next five 
years? Do teachers/ staff 
know and understand the 
vision?b) Who does your 
school consider to be your 
key stakeholders? How is 
this vision communicated 
to the overall school 
community?c) Who is 
involved in setting this 
vision/ strategy? When 
there is disagreement, how 
does the school leader 
build alignment 


¢ School either has no 
clear vision, or one 
defined without 
substantial stakeholder 
collaboration * School 
vision focuses primarily 
on meeting state/ 
national mandates* 
School leader does not or 
cannot articulate a clear 
focus on building an 
environment conducive to 
learning 


« The school has a vision 
statement, focused 
primarily on one aspect 
of schooling such as 
exams Teachers /parents 
/students /community 
have a weak 
understanding of the 
vision. 


* School has defined a 
vision that focuses on 
improvement in student 
outcomes, but largely 
focused on meeting 
state/ national mandatess 
School vision is usually 
defined with limited 
stakeholder 
collaborations School 
leaders may focus on the 
quality of the overall 
school environment, but 
often in response to 
specific issues 


* School has defined a 
vision that focuses on 
improvement in student 
outcomes, beyond just 
exam results* Teachers 
and parents are involved 
in defining this vision, 
which is somewhat 
responsive to local needs 


* School leaders define 
and broadly 
communicate a shared 
vision and purpose for 
the school that focuses 
on improving student 
learning (often beyond 
those required by law)s 
Vision and purpose is 
built upon a keen 
understanding of student 
and community needs, 
and defined 
collaboratively with a 
wide range of 
stakeholders* School 
leader proactively builds 
environment conducive to 
learning 


11. Clearly 
defined 
accountability for 
school leaders 


Tests whether 
school leaders are 
accountable for 
delivery of student 
outcomes 


a) Who is accountable for 
delivering on school 
targets? 

b) How are individual 
school leaders held 
responsible for the delivery 
of targets? Does this apply 
to equity and cost targets 
as well as quality targets? 
c) What authority do you 
have to impact factors that 
would allow them to meet 
those targets (e.g. 
budgetary authority, hiring 
& firing)? Is this sufficient? 


* School leaders are only 
held accountable for 
minimal targets (e.g. 
those set by 
government), without 
school- level or individual 
consequences for good 
and poor performance 

* School leaders have 
little or no autonomy to 
impact the areas of 
accountability 


* School leaders feel 
accountable for 
performance 

¢ There are some 
consequences for good 
and bad performance, 
but they may not be clear 
or consistently applied 

¢ There is a limited sense 
of personal responsibility 


* School leaders are held 
accountable for absolute 
number of student 
reaching targets set by 
government and school 
internally 

¢ There are some school- 
level and individual 
consequences for good 
and poor performance 

* School leaders are 
provided some autonomy 
to impact the areas of 
accountability 


* School leaders feel 
accountable for a range 
of outcomes, with clearly 
defined consequences 
for good and bad 
performance 


* School leaders are held 
accountable for quality, 
equity and cost- 
effectiveness of student 
outcomes within the 
school, with school-level 
and individual 
consequences for good 
and poor performance 

* School leaders are 
provided sufficient 
autonomy to impact the 
areas of accountability 
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Topic 


Questions 


Scoring Rubric 


3 


People Management 


12. People 
management 


Tests how well 
the school 
identifies and 
targets teaching 
needs, capacity 
gaps in the 
school, and how 
they find the right 
teachers 


a) Who makes decisions to 


hire teachers? 

b) Who determines the 
recruitment criteria? 

c) How do you know who 
are your best teachers? 
d) What types of rewards 
are offered to teachers? 
e) How do you keep the 
best teachers motivated? 
f) How do you know who 
are the teachers who are 
not doing so well? 


* The school has very 
limited or no control over 
recruitment of staff 
(teachers are assigned to 
the school) or there is 
discrimination in the 
recruitment process 

¢ There is limited or no 
structured way of 
monitoring the 
performance of teachers 


¢ The school has some 
control over recruitment, 
but there is no standard 
process 

¢ The school has some 
ways to address good or 
poor performance, but 
rarely in a structured way 


¢ Schools refer to 
centralised authorities to 
recruit and deploy 
teachers but can send 
requests for resourcing. 
¢ Performance 
management is dictated 
by centralised authorities 


* The school controls the 
number of teachers that 
are hired and has a clear 
interview process, which 
sometimes or 
occasionally goes 
beyond just interview 

¢ There is a teacher 
evaluation system that 
rewards good 
performance and 
evaluates teachers 
beyond exam scores 

* Poor teaching is 
addressed through 


* School proactively 
works with centralised 
authorities to recruit and 
retain the best teachers. 
* Teachers undergo 
formal and informal 
reviews regularly. 

* Motivational rewards 
are available in the 
school. 

* Poor teaching is 
addressed using a range 
of targeted interventions 
(e.g. team teaching, CPD 
training, performance 


limited range of appraisals) 
interventions (e.g. team 
teaching, CPD training, 
performance appraisals) 
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